{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "77e04b29-78d8-4619-a95a-4817a0fcc6a1",
   "metadata": {},
   "source": [
    "# 一、项目背景介绍\n",
    "该项目为推荐系统deepFM算法模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7f633c5-e91d-4593-a830-ef89aafedcdb",
   "metadata": {},
   "source": [
    "# 二、数据介绍\n",
    "数据集有用户ID,离散标签，连续标签和预测标签组成，如[ID,预测标签，I1,I2,..I13,C1,C2,...C26]\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/1fa1d2ab991a479e90c40e517181e91c065dbbd998dd4f53897d54cc0c5ec1dd)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3133c83f-4197-46eb-b120-4e033f12d0f0",
   "metadata": {},
   "source": [
    "# 模型介绍\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/c52751045c0c4a40a7da439bf957379251b8b230d98d4aae9cfc5428bf5b56b2)\n",
    "\n",
    "DeepFM由两部分组成:神经网络部分和因子分解部分，分别负责提取低阶特征和高阶特征。这两部分共享相同的输入。DeepFM的预测结果可以写成:\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/bc24106d65de471285fd3c2c5838bd6f067f28d37c14476ab4b86c09582b947b)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc7f9f3d-328e-42b1-a3b1-9c6688c19b57",
   "metadata": {},
   "source": [
    "# 四、模型训练\n",
    "\n",
    "(由于时间太赶，代码还没排版好，下次有时间再重新整理一下代码)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "24e6da50-2ffa-4cfb-b21d-cac646d108ee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:13:27.244919Z",
     "iopub.status.busy": "2022-03-06T07:13:27.243809Z",
     "iopub.status.idle": "2022-03-06T07:13:28.151134Z",
     "shell.execute_reply": "2022-03-06T07:13:28.150409Z",
     "shell.execute_reply.started": "2022-03-06T07:13:27.244880Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\r\n",
    "import numpy as np\r\n",
    "import pandas as pd\r\n",
    "import matplotlib.pyplot as plt\r\n",
    "from tqdm import tqdm\r\n",
    "import datetime\r\n",
    "\r\n",
    "import warnings\r\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aef54136-5992-44e4-8f31-48c8c5da3f9f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:14:27.636544Z",
     "iopub.status.busy": "2022-03-06T07:14:27.635609Z",
     "iopub.status.idle": "2022-03-06T07:14:27.640022Z",
     "shell.execute_reply": "2022-03-06T07:14:27.639335Z",
     "shell.execute_reply.started": "2022-03-06T07:14:27.636451Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_path=\"./preprocess data/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "caeec8a7-91e3-471f-a98e-4040eb7f8244",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:14:47.033986Z",
     "iopub.status.busy": "2022-03-06T07:14:47.033110Z",
     "iopub.status.idle": "2022-03-06T07:14:47.039954Z",
     "shell.execute_reply": "2022-03-06T07:14:47.039319Z",
     "shell.execute_reply.started": "2022-03-06T07:14:47.033947Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def prepared_data(file_path):\r\n",
    "    # 读入训练集， 验证集和测试集\r\n",
    "    train = pd.read_csv(file_path + 'train_set.csv')\r\n",
    "    val = pd.read_csv(file_path + 'val_set.csv')\r\n",
    "    test = pd.read_csv(file_path + 'test_set.csv')\r\n",
    "    \r\n",
    "    trn_x, trn_y = train.drop(columns='Label').values, train['Label'].values\r\n",
    "    val_x, val_y = val.drop(columns='Label').values, val['Label'].values\r\n",
    "    test_x = test.values\r\n",
    "    \r\n",
    "    fea_col = np.load(file_path + 'fea_col.npy', allow_pickle=True)\r\n",
    "    \r\n",
    "    return fea_col, (trn_x, trn_y), (val_x, val_y), test_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6a929854-3823-496a-a937-5e9543fbc416",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:15:20.688514Z",
     "iopub.status.busy": "2022-03-06T07:15:20.687330Z",
     "iopub.status.idle": "2022-03-06T07:15:20.718152Z",
     "shell.execute_reply": "2022-03-06T07:15:20.717210Z",
     "shell.execute_reply.started": "2022-03-06T07:15:20.688452Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fea_cols, (trn_x, trn_y), (val_x, val_y), test_x = prepared_data(data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e11be868-cc0b-482a-b19e-caea4d74fd36",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:17:49.510683Z",
     "iopub.status.busy": "2022-03-06T07:17:49.509966Z",
     "iopub.status.idle": "2022-03-06T07:17:49.518845Z",
     "shell.execute_reply": "2022-03-06T07:17:49.518206Z",
     "shell.execute_reply.started": "2022-03-06T07:17:49.510645Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([{'feat': 'I1'},\n",
       "  {'feat': 'I2'},\n",
       "  {'feat': 'I3'},\n",
       "  {'feat': 'I4'},\n",
       "  {'feat': 'I5'},\n",
       "  {'feat': 'I6'},\n",
       "  {'feat': 'I7'},\n",
       "  {'feat': 'I8'},\n",
       "  {'feat': 'I9'},\n",
       "  {'feat': 'I10'},\n",
       "  {'feat': 'I11'},\n",
       "  {'feat': 'I12'},\n",
       "  {'feat': 'I13'}],\n",
       " [{'feat': 'C1', 'feat_num': 79, 'embed_dim': 8},\n",
       "  {'feat': 'C2', 'feat_num': 252, 'embed_dim': 8},\n",
       "  {'feat': 'C3', 'feat_num': 1293, 'embed_dim': 8},\n",
       "  {'feat': 'C4', 'feat_num': 1043, 'embed_dim': 8},\n",
       "  {'feat': 'C5', 'feat_num': 30, 'embed_dim': 8},\n",
       "  {'feat': 'C6', 'feat_num': 7, 'embed_dim': 8},\n",
       "  {'feat': 'C7', 'feat_num': 1164, 'embed_dim': 8},\n",
       "  {'feat': 'C8', 'feat_num': 39, 'embed_dim': 8},\n",
       "  {'feat': 'C9', 'feat_num': 2, 'embed_dim': 8},\n",
       "  {'feat': 'C10', 'feat_num': 908, 'embed_dim': 8},\n",
       "  {'feat': 'C11', 'feat_num': 926, 'embed_dim': 8},\n",
       "  {'feat': 'C12', 'feat_num': 1239, 'embed_dim': 8},\n",
       "  {'feat': 'C13', 'feat_num': 824, 'embed_dim': 8},\n",
       "  {'feat': 'C14', 'feat_num': 20, 'embed_dim': 8},\n",
       "  {'feat': 'C15', 'feat_num': 819, 'embed_dim': 8},\n",
       "  {'feat': 'C16', 'feat_num': 1159, 'embed_dim': 8},\n",
       "  {'feat': 'C17', 'feat_num': 9, 'embed_dim': 8},\n",
       "  {'feat': 'C18', 'feat_num': 534, 'embed_dim': 8},\n",
       "  {'feat': 'C19', 'feat_num': 201, 'embed_dim': 8},\n",
       "  {'feat': 'C20', 'feat_num': 4, 'embed_dim': 8},\n",
       "  {'feat': 'C21', 'feat_num': 1204, 'embed_dim': 8},\n",
       "  {'feat': 'C22', 'feat_num': 7, 'embed_dim': 8},\n",
       "  {'feat': 'C23', 'feat_num': 12, 'embed_dim': 8},\n",
       "  {'feat': 'C24', 'feat_num': 729, 'embed_dim': 8},\n",
       "  {'feat': 'C25', 'feat_num': 33, 'embed_dim': 8},\n",
       "  {'feat': 'C26', 'feat_num': 554, 'embed_dim': 8}])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fea_cols[0],fea_cols[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f107daac-9d67-42f4-8946-5b54f331f545",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:27:57.412746Z",
     "iopub.status.busy": "2022-03-06T07:27:57.411871Z",
     "iopub.status.idle": "2022-03-06T07:27:58.912413Z",
     "shell.execute_reply": "2022-03-06T07:27:58.911406Z",
     "shell.execute_reply.started": "2022-03-06T07:27:57.412704Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import paddle\r\n",
    "import paddle.nn as nn\r\n",
    "from paddle.io import Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "761c89d1-e5be-431c-8254-c62f52c96090",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:28:02.281775Z",
     "iopub.status.busy": "2022-03-06T07:28:02.281151Z",
     "iopub.status.idle": "2022-03-06T07:28:02.289134Z",
     "shell.execute_reply": "2022-03-06T07:28:02.288226Z",
     "shell.execute_reply.started": "2022-03-06T07:28:02.281732Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#构建数据集\r\n",
    "\r\n",
    "# 自定义数据集\r\n",
    "#映射式(map-style)数据集需要继承paddle.io.Dataset\r\n",
    "class SelfDefinedDataset(Dataset):\r\n",
    "    def __init__(self, data_x, data_y, mode = 'train'):\r\n",
    "        super(SelfDefinedDataset, self).__init__()\r\n",
    "        self.data_x = data_x\r\n",
    "        self.data_y = data_y\r\n",
    "        self.mode = mode\r\n",
    "\r\n",
    "    def __getitem__(self, idx):\r\n",
    "        if self.mode == 'predict':\r\n",
    "           return self.data_x[idx]\r\n",
    "        else:\r\n",
    "           return self.data_x[idx], self.data_y[idx]\r\n",
    "\r\n",
    "    def __len__(self):\r\n",
    "        return len(self.data_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "99ff735b-df7b-42dc-8a01-c4a036a434ee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:33:18.652268Z",
     "iopub.status.busy": "2022-03-06T07:33:18.651396Z",
     "iopub.status.idle": "2022-03-06T07:33:18.655891Z",
     "shell.execute_reply": "2022-03-06T07:33:18.655234Z",
     "shell.execute_reply.started": "2022-03-06T07:33:18.652230Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dl_train_dataset=SelfDefinedDataset(trn_x, trn_y)\r\n",
    "dl_val_dataset=SelfDefinedDataset(val_x, val_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f6350230-485b-41db-93ab-b024c1f7daed",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:34:09.338566Z",
     "iopub.status.busy": "2022-03-06T07:34:09.337380Z",
     "iopub.status.idle": "2022-03-06T07:34:09.344571Z",
     "shell.execute_reply": "2022-03-06T07:34:09.343749Z",
     "shell.execute_reply.started": "2022-03-06T07:34:09.338520Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(39,) ()\n",
      "[9.47368421e-02 3.81388253e-04 8.27716684e-04 8.04597701e-02\n",
      " 1.21156602e-04 1.50927124e-03 5.42822678e-03 9.14076782e-03\n",
      " 1.24179528e-03 2.50000000e-01 2.70270270e-02 0.00000000e+00\n",
      " 9.09090909e-03 2.60000000e+01 1.20000000e+01 7.50000000e+01\n",
      " 3.93000000e+02 1.00000000e+00 0.00000000e+00 9.69000000e+02\n",
      " 2.70000000e+01 1.00000000e+00 8.55000000e+02 3.32000000e+02\n",
      " 6.59000000e+02 5.09000000e+02 1.30000000e+01 2.73000000e+02\n",
      " 2.65000000e+02 8.00000000e+00 1.88000000e+02 2.30000000e+01\n",
      " 2.00000000e+00 1.03300000e+03 0.00000000e+00 0.00000000e+00\n",
      " 6.70000000e+01 2.70000000e+01 4.25000000e+02] 0\n"
     ]
    }
   ],
   "source": [
    "for data, label in dl_train_dataset:\r\n",
    "    print(data.shape, label.shape)\r\n",
    "    print(data, label)\r\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6212670b-8455-445a-8905-02223f85e711",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:36:13.723777Z",
     "iopub.status.busy": "2022-03-06T07:36:13.722626Z",
     "iopub.status.idle": "2022-03-06T07:36:13.727984Z",
     "shell.execute_reply": "2022-03-06T07:36:13.727301Z",
     "shell.execute_reply.started": "2022-03-06T07:36:13.723736Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_loader = paddle.io.DataLoader(dl_train_dataset, batch_size = 32, shuffle = True)\r\n",
    "valid_loader = paddle.io.DataLoader(dl_val_dataset, batch_size = 32, shuffle = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5ffc1170-60c8-4f47-8acf-0f0983d50069",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T07:37:15.469170Z",
     "iopub.status.busy": "2022-03-06T07:37:15.468426Z",
     "iopub.status.idle": "2022-03-06T07:37:15.481342Z",
     "shell.execute_reply": "2022-03-06T07:37:15.480550Z",
     "shell.execute_reply.started": "2022-03-06T07:37:15.469129Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[32, 39] Tensor(shape=[32], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,\n",
      "        0, 0, 0, 0, 1, 1, 0, 0])\n"
     ]
    }
   ],
   "source": [
    "\r\n",
    "for x, y in iter(train_loader):\r\n",
    "    print(x.shape, y)\r\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4d21a5d-bcf7-4c9c-8ecf-20b693e56781",
   "metadata": {},
   "source": [
    "# 构建模型\n",
    "这个模型是两部分组成， 左边是FM模型， 右边是DNN模型\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/a1b104bf5d3e42dd83280daf98f5741217ce81455ae34cc1ba9015b5a7fdccac)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "621cef26-fd1a-422f-9a46-788fefde2f60",
   "metadata": {},
   "source": [
    "##FM公式\n",
    "(![](https://ai-studio-static-online.cdn.bcebos.com/256411e6caa2444b9408dabc45686f8169e8be0fe60e41c1b9cb4eba2554b772))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "8b57319e-428b-477e-ad67-38a130cfae45",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T14:42:52.477170Z",
     "iopub.status.busy": "2022-03-06T14:42:52.476421Z",
     "iopub.status.idle": "2022-03-06T14:42:52.485770Z",
     "shell.execute_reply": "2022-03-06T14:42:52.485069Z",
     "shell.execute_reply.started": "2022-03-06T14:42:52.477128Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "class FM(nn.Layer):\r\n",
    "    def __init__(self,dim,fea_num):\r\n",
    "        super(FM,self).__init__()\r\n",
    "        self.dim = dim\r\n",
    "        self.w0 = self.create_parameter(paddle.zeros(shape=[1,]).shape)\r\n",
    "        self.add_parameter('w0',self.w0)\r\n",
    "        self.w1=self.create_parameter(paddle.rand(shape=[fea_num, 1]).shape)\r\n",
    "        self.add_parameter('w1',self.w1)\r\n",
    "        self.w2 = self.create_parameter(paddle.rand(shape=[fea_num, dim]).shape)\r\n",
    "        self.add_parameter('w2',self.w2)\r\n",
    "        \r\n",
    "        \r\n",
    "\r\n",
    "\r\n",
    "    def forward(self,inputs):\r\n",
    "        \r\n",
    "        first_order=self.w0+paddle.mm(inputs,self.w1)\r\n",
    "        second_order = 1/2 * paddle.sum(\r\n",
    "            paddle.pow(paddle.mm(inputs, self.w2), 2) - paddle.mm(paddle.pow(inputs,2), paddle.pow(self.w2, 2)),\r\n",
    "            axis = 1,\r\n",
    "            keepdim = True\r\n",
    "        )\r\n",
    "\r\n",
    "        return first_order + second_order\r\n",
    "       \r\n",
    "        \r\n",
    "\r\n",
    "        \r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "8bae2d8a-ccba-4090-a011-582fced0a4c0",
   "metadata": {
    "collapsed": true,
    "execution": {
     "iopub.execute_input": "2022-03-06T13:35:40.111991Z",
     "iopub.status.busy": "2022-03-06T13:35:40.111101Z",
     "iopub.status.idle": "2022-03-06T13:35:40.136109Z",
     "shell.execute_reply": "2022-03-06T13:35:40.135289Z",
     "shell.execute_reply.started": "2022-03-06T13:35:40.111954Z"
    },
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------\n",
      " Layer (type)       Input Shape          Output Shape         Param #    \n",
      "===========================================================================\n",
      "     FM-18           [[32, 39]]            [32, 1]              352      \n",
      "===========================================================================\n",
      "Total params: 352\n",
      "Trainable params: 352\n",
      "Non-trainable params: 0\n",
      "---------------------------------------------------------------------------\n",
      "Input size (MB): 0.00\n",
      "Forward/backward pass size (MB): 0.00\n",
      "Params size (MB): 0.00\n",
      "Estimated Total Size (MB): 0.01\n",
      "---------------------------------------------------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'total_params': 352, 'trainable_params': 352}"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paddle.summary(FM(8,39),(32,39))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "bec2a36c-619c-4d59-bf97-2cef5d0f7aff",
   "metadata": {
    "collapsed": true,
    "execution": {
     "iopub.execute_input": "2022-03-06T13:42:42.183554Z",
     "iopub.status.busy": "2022-03-06T13:42:42.182678Z",
     "iopub.status.idle": "2022-03-06T13:42:42.193915Z",
     "shell.execute_reply": "2022-03-06T13:42:42.193300Z",
     "shell.execute_reply.started": "2022-03-06T13:42:42.183517Z"
    },
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w0 Parameter containing:\n",
      "Tensor(shape=[1], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
      "       [0.44599235])\n",
      "w1 Parameter containing:\n",
      "Tensor(shape=[39, 1], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
      "       [[ 0.15212154],\n",
      "        [-0.23500401],\n",
      "        [ 0.15196240],\n",
      "        [-0.24409188],\n",
      "        [ 0.33634973],\n",
      "        [-0.25536168],\n",
      "        [-0.38496849],\n",
      "        [ 0.12309468],\n",
      "        [ 0.08477345],\n",
      "        [-0.28101498],\n",
      "        [-0.31306419],\n",
      "        [-0.24703261],\n",
      "        [-0.32349095],\n",
      "        [ 0.09884590],\n",
      "        [ 0.10089105],\n",
      "        [-0.16122256],\n",
      "        [ 0.37944883],\n",
      "        [-0.38282561],\n",
      "        [-0.25189462],\n",
      "        [ 0.23034543],\n",
      "        [ 0.28095150],\n",
      "        [-0.19442679],\n",
      "        [ 0.31634802],\n",
      "        [ 0.37122297],\n",
      "        [-0.00535947],\n",
      "        [ 0.02663028],\n",
      "        [-0.33284673],\n",
      "        [ 0.20504153],\n",
      "        [ 0.14450914],\n",
      "        [ 0.30093455],\n",
      "        [ 0.30281061],\n",
      "        [ 0.35462838],\n",
      "        [ 0.23657298],\n",
      "        [-0.07216120],\n",
      "        [ 0.24004394],\n",
      "        [ 0.26921648],\n",
      "        [ 0.03911385],\n",
      "        [-0.20583311],\n",
      "        [ 0.28365976]])\n",
      "w2 Parameter containing:\n",
      "Tensor(shape=[39, 8], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
      "       [[ 0.15577620,  0.22880781,  0.27651924,  0.10041943,  0.32429308,\n",
      "         -0.20779081,  0.32520187,  0.32720041],\n",
      "        [ 0.07708156,  0.06736505, -0.30282313, -0.25281292, -0.29670084,\n",
      "         -0.19217210,  0.01393062, -0.00345838],\n",
      "        [-0.17628576, -0.03204566, -0.07942885, -0.20643008,  0.28758377,\n",
      "          0.11286438, -0.31433737,  0.32690459],\n",
      "        [-0.00155842, -0.30041787,  0.26664633, -0.34305304,  0.21127844,\n",
      "         -0.02126777, -0.10404870, -0.27173924],\n",
      "        [ 0.07024291, -0.01881120,  0.31316739, -0.34119892, -0.16596176,\n",
      "          0.27427781,  0.23088801,  0.12650532],\n",
      "        [-0.31173509,  0.14822412, -0.21060841,  0.07805938,  0.05357620,\n",
      "         -0.22155873,  0.13462356, -0.23946336],\n",
      "        [-0.07896140, -0.09784833,  0.22657692, -0.10365465, -0.10402825,\n",
      "         -0.25557119, -0.27888602,  0.22910529],\n",
      "        [-0.02517712, -0.21645448, -0.21438505,  0.08983538, -0.09465221,\n",
      "         -0.19724530, -0.10479179, -0.32254085],\n",
      "        [-0.24668950, -0.15644391,  0.17139834,  0.34450704, -0.09571847,\n",
      "         -0.08394209,  0.16589701,  0.04212463],\n",
      "        [ 0.02561194,  0.14452714,  0.14460653,  0.18652892,  0.04403603,\n",
      "         -0.10946128, -0.20260842, -0.15948185],\n",
      "        [ 0.08319396, -0.07356673, -0.32240659, -0.14773907,  0.23563683,\n",
      "         -0.24239624,  0.02249771,  0.18951678],\n",
      "        [ 0.27436978, -0.29182184, -0.31366554, -0.33641908,  0.25853056,\n",
      "          0.34484267,  0.34796137, -0.33239374],\n",
      "        [-0.16921400, -0.31563374,  0.17729062, -0.35107684,  0.15638494,\n",
      "         -0.01972738, -0.04920489,  0.21782851],\n",
      "        [ 0.25486714, -0.21163081,  0.33196145, -0.25268283,  0.27995855,\n",
      "         -0.15205613,  0.12195754, -0.28402835],\n",
      "        [-0.04137045, -0.34036902,  0.06647977,  0.09346011,  0.07135770,\n",
      "         -0.31830025,  0.15004754, -0.01303279],\n",
      "        [ 0.02361777,  0.23501348, -0.21050361,  0.28536940, -0.01585212,\n",
      "         -0.18341446, -0.30547366,  0.05710134],\n",
      "        [ 0.27275360,  0.05684346,  0.27674222,  0.25651270, -0.19693002,\n",
      "         -0.04147559, -0.07746217,  0.29535502],\n",
      "        [ 0.01637539, -0.01279199, -0.34297699,  0.12323886, -0.24553974,\n",
      "          0.15270334, -0.33256617, -0.26800191],\n",
      "        [ 0.11013299,  0.14546180,  0.21839547,  0.27261269,  0.21981508,\n",
      "         -0.35508502,  0.30041027,  0.19635594],\n",
      "        [-0.03015390,  0.22929645, -0.08106649,  0.33801877,  0.30294812,\n",
      "          0.14668328,  0.22728223, -0.12542756],\n",
      "        [ 0.11531752,  0.24197215,  0.09719422,  0.06384772, -0.02611512,\n",
      "          0.27705836,  0.26074481, -0.09289709],\n",
      "        [-0.00383213,  0.20810956,  0.16478932,  0.32767135,  0.33111262,\n",
      "          0.01151341,  0.32297587, -0.21090922],\n",
      "        [ 0.29925239,  0.02241808,  0.13874969,  0.00231245, -0.07995850,\n",
      "         -0.18352450,  0.09174320, -0.19485711],\n",
      "        [-0.24509637,  0.27009863, -0.01161408,  0.20424527, -0.13978456,\n",
      "         -0.29609233,  0.31746858,  0.04315117],\n",
      "        [ 0.27329719, -0.12529290, -0.32698318, -0.24654844,  0.23167288,\n",
      "         -0.15427460,  0.10228488, -0.17887318],\n",
      "        [-0.20085220, -0.00409412,  0.25018805, -0.22700429,  0.29467630,\n",
      "         -0.15312642,  0.24168700,  0.05744749],\n",
      "        [-0.00520149,  0.24608541, -0.27737081, -0.11545230, -0.09527269,\n",
      "          0.00870335, -0.18073319, -0.08833805],\n",
      "        [ 0.28675926,  0.24205035, -0.05663669,  0.34334344, -0.14382899,\n",
      "          0.00349540,  0.09571490,  0.17503434],\n",
      "        [-0.25846153,  0.27690506, -0.02511346, -0.16538288, -0.15035672,\n",
      "          0.05361632,  0.28250009,  0.03318775],\n",
      "        [-0.29530585,  0.11635727, -0.03440252, -0.02936518, -0.12455243,\n",
      "          0.22812080,  0.14757043, -0.01553726],\n",
      "        [ 0.02676344,  0.06184769, -0.32930714, -0.25325581,  0.27421319,\n",
      "         -0.35177410, -0.06330565,  0.18976521],\n",
      "        [-0.26285112, -0.34553891,  0.20709741, -0.04179189,  0.24602813,\n",
      "          0.10477316,  0.16264081, -0.26961526],\n",
      "        [-0.01329854, -0.10599869,  0.30741388, -0.20692165,  0.15986443,\n",
      "         -0.32144365, -0.13574407,  0.25963974],\n",
      "        [-0.29203293,  0.16864133, -0.05458003, -0.30778402,  0.16307658,\n",
      "          0.25062662,  0.24577838,  0.29097217],\n",
      "        [ 0.29053229,  0.22855347, -0.33323759, -0.06624693,  0.28163993,\n",
      "         -0.30149159,  0.09431383, -0.22111763],\n",
      "        [-0.31673843,  0.33145469,  0.04070696, -0.18406132, -0.02175492,\n",
      "         -0.03366020,  0.34052783,  0.05786243],\n",
      "        [-0.23167378, -0.13213293, -0.11643942,  0.35722369, -0.01391965,\n",
      "          0.34539843,  0.09799147,  0.31985027],\n",
      "        [-0.07074884, -0.11628295,  0.16826475,  0.01482019, -0.18034782,\n",
      "          0.13993672, -0.09274089, -0.34356785],\n",
      "        [ 0.34847683, -0.34235564, -0.18613826, -0.10193998, -0.12325345,\n",
      "         -0.33056688, -0.01712272, -0.17934202]])\n"
     ]
    }
   ],
   "source": [
    "s=FM(8,39)\r\n",
    "for name, param in s.named_parameters():\r\n",
    "    print(name, param)      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c7c9a80-9614-42e6-9dbf-bc2c9c2e9b8b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "799f43de-9239-4382-8402-909aebc46217",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T13:56:34.294229Z",
     "iopub.status.busy": "2022-03-06T13:56:34.293317Z",
     "iopub.status.idle": "2022-03-06T13:56:34.297792Z",
     "shell.execute_reply": "2022-03-06T13:56:34.297173Z",
     "shell.execute_reply.started": "2022-03-06T13:56:34.294191Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import paddle.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "c13e66ab-8f98-497c-87d0-4f4f26c5ea50",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T14:11:14.058784Z",
     "iopub.status.busy": "2022-03-06T14:11:14.058317Z",
     "iopub.status.idle": "2022-03-06T14:11:14.065696Z",
     "shell.execute_reply": "2022-03-06T14:11:14.064917Z",
     "shell.execute_reply.started": "2022-03-06T14:11:14.058746Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "class Dnn(nn.Layer):\r\n",
    "    def __init__(self,hidden_list,dropout=0.):\r\n",
    "        super(Dnn,self).__init__()\r\n",
    "        self.dnn=nn.LayerList([nn.Linear(layer[0],layer[1]) \r\n",
    "          for layer in  list(zip(hidden_list[:-1],hidden_list[1:]))])\r\n",
    "        self.dropout=nn.Dropout(dropout)\r\n",
    "        \r\n",
    "\r\n",
    "    def forward(self,x):\r\n",
    "        for linear in self.dnn:\r\n",
    "            x=linear(x)\r\n",
    "            x=F.relu(x)\r\n",
    "\r\n",
    "        self.dropout(x)\r\n",
    "        \r\n",
    "        \r\n",
    "        return(x)\r\n",
    "        \r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "a19302d5-d29d-49ba-9bf8-7df3492ba433",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T14:12:22.082729Z",
     "iopub.status.busy": "2022-03-06T14:12:22.081633Z",
     "iopub.status.idle": "2022-03-06T14:12:22.093345Z",
     "shell.execute_reply": "2022-03-06T14:12:22.092621Z",
     "shell.execute_reply.started": "2022-03-06T14:12:22.082682Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------\n",
      " Layer (type)       Input Shape          Output Shape         Param #    \n",
      "===========================================================================\n",
      "   Linear-28        [[32, 128]]            [32, 64]            8,256     \n",
      "   Linear-29         [[32, 64]]            [32, 32]            2,080     \n",
      "   Dropout-2         [[32, 32]]            [32, 32]              0       \n",
      "===========================================================================\n",
      "Total params: 10,336\n",
      "Trainable params: 10,336\n",
      "Non-trainable params: 0\n",
      "---------------------------------------------------------------------------\n",
      "Input size (MB): 0.02\n",
      "Forward/backward pass size (MB): 0.03\n",
      "Params size (MB): 0.04\n",
      "Estimated Total Size (MB): 0.09\n",
      "---------------------------------------------------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'total_params': 10336, 'trainable_params': 10336}"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paddle.summary(Dnn([128, 64, 32]),(32,128))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "be4fafe7-262d-4c34-83db-8283e2f23dbc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T14:13:22.524194Z",
     "iopub.status.busy": "2022-03-06T14:13:22.523721Z",
     "iopub.status.idle": "2022-03-06T14:13:22.535869Z",
     "shell.execute_reply": "2022-03-06T14:13:22.535078Z",
     "shell.execute_reply.started": "2022-03-06T14:13:22.524157Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dnn.0.weight Parameter containing:\n",
      "Tensor(shape=[128, 64], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
      "       [[ 0.06707515,  0.13847785, -0.06655482, ..., -0.16561088,\n",
      "         -0.06180841,  0.15573387],\n",
      "        [-0.09976895, -0.08881859, -0.02656919, ..., -0.05318263,\n",
      "         -0.17018560,  0.15193240],\n",
      "        [-0.00428711, -0.02451752,  0.16817419, ..., -0.05110016,\n",
      "         -0.01681779, -0.11648417],\n",
      "        ...,\n",
      "        [ 0.09143643, -0.09313656, -0.12042940, ...,  0.13972722,\n",
      "         -0.04550512, -0.05906151],\n",
      "        [-0.09089268, -0.04715914, -0.14746152, ..., -0.07574488,\n",
      "         -0.10247044,  0.17553581],\n",
      "        [ 0.04491828, -0.04316257,  0.10424592, ..., -0.13233250,\n",
      "         -0.07001856, -0.17624608]])\n",
      "dnn.0.bias Parameter containing:\n",
      "Tensor(shape=[64], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
      "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n",
      "dnn.1.weight Parameter containing:\n",
      "Tensor(shape=[64, 32], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
      "       [[ 0.16170856,  0.06038851,  0.03602764, ...,  0.12482542,\n",
      "         -0.02523838, -0.20853551],\n",
      "        [ 0.01870546, -0.12472783, -0.03738679, ..., -0.13065925,\n",
      "         -0.05682845, -0.04650307],\n",
      "        [ 0.23001781, -0.01276527, -0.02761041, ...,  0.06322876,\n",
      "         -0.09978248,  0.12670732],\n",
      "        ...,\n",
      "        [-0.01610865, -0.04137805, -0.16878791, ...,  0.10086116,\n",
      "          0.05140889, -0.07785009],\n",
      "        [ 0.05162212, -0.03805579,  0.06823462, ...,  0.14723462,\n",
      "         -0.18079299,  0.02479511],\n",
      "        [-0.18927671,  0.13601407,  0.10843480, ...,  0.20625535,\n",
      "          0.19575387,  0.09843263]])\n",
      "dnn.1.bias Parameter containing:\n",
      "Tensor(shape=[32], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
      "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "z=Dnn([128, 64, 32])\r\n",
    "for name, param in z.named_parameters():\r\n",
    "    print(name, param)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "id": "897a2954-96a0-482b-9cbe-c31cc6a2c805",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:32:48.476591Z",
     "iopub.status.busy": "2022-03-06T17:32:48.475715Z",
     "iopub.status.idle": "2022-03-06T17:32:48.500440Z",
     "shell.execute_reply": "2022-03-06T17:32:48.499192Z",
     "shell.execute_reply.started": "2022-03-06T17:32:48.476523Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "class DeepFM(nn.Layer):\r\n",
    "    def __init__(self,feature_columns,hidden_list,dnn_dropout=0.):\r\n",
    "        super(DeepFM,self).__init__()\r\n",
    "        self.dense_feature_cols, self.sparse_feature_cols = feature_columns\r\n",
    "        \r\n",
    "        #embedding\r\n",
    "        self.embed_layers=nn.LayerDict({\r\n",
    "            \"embed_\"+str(i):nn.Embedding(feat['feat_num'],feat['embed_dim'])   for i,feat in enumerate(self.sparse_feature_cols)\r\n",
    "        })\r\n",
    "        \r\n",
    "        \r\n",
    "        \r\n",
    "        self.fea_num=len(self.dense_feature_cols)+len(self.sparse_feature_cols)*self.sparse_feature_cols[0]['embed_dim']\r\n",
    "        hidden_list.insert(0,self.fea_num)\r\n",
    "        self.fm=FM(self.sparse_feature_cols[0]['embed_dim'],self.fea_num)\r\n",
    "        self.dnn_network=Dnn(hidden_list,dnn_dropout)\r\n",
    "        self.nn_final_linear=nn.Linear(hidden_list[-1],1)\r\n",
    "        \r\n",
    "        \r\n",
    "        \r\n",
    "    def forward(self,x):\r\n",
    "        dense_inputs,sparse_inputs = x[:,:len(self.dense_feature_cols)],x[:,len(self.dense_feature_cols):]\r\n",
    "        sparse_inputs = paddle.cast(sparse_inputs, dtype='long')\r\n",
    "        sparse_embeds=[self.embed_layers['embed_'+str(i)](sparse_inputs[:,i])  for i  in range(26)]\r\n",
    "        # sparse_embeds=[self.embed_layers['embed_'+str(i)](sparse_inputs[:,i])  for i  in range(sparse_inputs.shape[1])]\r\n",
    "        sparse_embeds=paddle.concat(sparse_embeds,axis=-1)\r\n",
    "        \r\n",
    "        #把离散特征和连续特征进行拼接作为FM和DNN的输入\r\n",
    "        dense_inputs = paddle.cast(dense_inputs, dtype='float32')\r\n",
    "        sparse_inputs = paddle.cast(sparse_inputs, dtype='float32')\r\n",
    "        x=paddle.concat([sparse_embeds,dense_inputs],axis=-1)\r\n",
    "        #wide\r\n",
    "        wide_outputs=self.fm(x)\r\n",
    "        #deep\r\n",
    "        deep_outputs=self.nn_final_linear(self.dnn_network(x))\r\n",
    "        \r\n",
    "        #模型的输出\r\n",
    "        outputs=F.sigmoid(paddle.add(wide_outputs,deep_outputs))\r\n",
    "        \r\n",
    "        return outputs\r\n",
    "        \r\n",
    "        \r\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "id": "028ba657-c2f3-4b32-b4ea-72b2f01ff36c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:32:49.434721Z",
     "iopub.status.busy": "2022-03-06T17:32:49.434084Z",
     "iopub.status.idle": "2022-03-06T17:32:49.467278Z",
     "shell.execute_reply": "2022-03-06T17:32:49.466284Z",
     "shell.execute_reply.started": "2022-03-06T17:32:49.434653Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 建立模型\r\n",
    "hidden_list = [128, 64, 32]\r\n",
    "dnn_dropout = 0.\r\n",
    "\r\n",
    "model = DeepFM(fea_cols, hidden_list, dnn_dropout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "id": "adb282e7-5d50-4e5c-977a-0ea52adeaf69",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:32:50.078706Z",
     "iopub.status.busy": "2022-03-06T17:32:50.078024Z",
     "iopub.status.idle": "2022-03-06T17:32:50.101189Z",
     "shell.execute_reply": "2022-03-06T17:32:50.099980Z",
     "shell.execute_reply.started": "2022-03-06T17:32:50.078633Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(shape=[32, 1], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
      "       [[0.32315740],\n",
      "        [0.31093273],\n",
      "        [0.16905159],\n",
      "        [0.19599071],\n",
      "        [0.29962289],\n",
      "        [0.29254788],\n",
      "        [0.33215153],\n",
      "        [0.24165253],\n",
      "        [0.26359463],\n",
      "        [0.24375126],\n",
      "        [0.27597246],\n",
      "        [0.26628450],\n",
      "        [0.20531006],\n",
      "        [0.26816389],\n",
      "        [0.24701035],\n",
      "        [0.23330608],\n",
      "        [0.22911505],\n",
      "        [0.22370060],\n",
      "        [0.24913113],\n",
      "        [0.22915682],\n",
      "        [0.19512792],\n",
      "        [0.27199194],\n",
      "        [0.21131818],\n",
      "        [0.18487053],\n",
      "        [0.26939511],\n",
      "        [0.22719760],\n",
      "        [0.29795116],\n",
      "        [0.20505209],\n",
      "        [0.23758069],\n",
      "        [0.17697102],\n",
      "        [0.23378713],\n",
      "        [0.27061450]])\n"
     ]
    }
   ],
   "source": [
    "for fea,label in iter(train_loader):\r\n",
    "    out=model(fea)\r\n",
    "    print(out)\r\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "id": "4b9ceefd-8f36-4fac-9fbf-ad5e3932fd16",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:33:08.572467Z",
     "iopub.status.busy": "2022-03-06T17:33:08.571430Z",
     "iopub.status.idle": "2022-03-06T17:33:08.593974Z",
     "shell.execute_reply": "2022-03-06T17:33:08.593080Z",
     "shell.execute_reply.started": "2022-03-06T17:33:08.572395Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------\n",
      " Layer (type)       Input Shape          Output Shape         Param #    \n",
      "===========================================================================\n",
      " Embedding-445         [[39]]              [39, 8]              632      \n",
      " Embedding-446         [[39]]              [39, 8]             2,016     \n",
      " Embedding-447         [[39]]              [39, 8]            10,344     \n",
      " Embedding-448         [[39]]              [39, 8]             8,344     \n",
      " Embedding-449         [[39]]              [39, 8]              240      \n",
      " Embedding-450         [[39]]              [39, 8]              56       \n",
      " Embedding-451         [[39]]              [39, 8]             9,312     \n",
      " Embedding-452         [[39]]              [39, 8]              312      \n",
      " Embedding-453         [[39]]              [39, 8]              16       \n",
      " Embedding-454         [[39]]              [39, 8]             7,264     \n",
      " Embedding-455         [[39]]              [39, 8]             7,408     \n",
      " Embedding-456         [[39]]              [39, 8]             9,912     \n",
      " Embedding-457         [[39]]              [39, 8]             6,592     \n",
      " Embedding-458         [[39]]              [39, 8]              160      \n",
      " Embedding-459         [[39]]              [39, 8]             6,552     \n",
      " Embedding-460         [[39]]              [39, 8]             9,272     \n",
      " Embedding-461         [[39]]              [39, 8]              72       \n",
      " Embedding-462         [[39]]              [39, 8]             4,272     \n",
      " Embedding-463         [[39]]              [39, 8]             1,608     \n",
      " Embedding-464         [[39]]              [39, 8]              32       \n",
      " Embedding-465         [[39]]              [39, 8]             9,632     \n",
      " Embedding-466         [[39]]              [39, 8]              56       \n",
      " Embedding-467         [[39]]              [39, 8]              96       \n",
      " Embedding-468         [[39]]              [39, 8]             5,832     \n",
      " Embedding-469         [[39]]              [39, 8]              264      \n",
      " Embedding-470         [[39]]              [39, 8]             4,432     \n",
      "     FM-37          [[39, 221]]            [39, 1]             1,990     \n",
      "  Linear-100        [[39, 221]]           [39, 128]           28,416     \n",
      "  Linear-101        [[39, 128]]            [39, 64]            8,256     \n",
      "  Linear-102         [[39, 64]]            [39, 32]            2,080     \n",
      "  Dropout-21         [[39, 32]]            [39, 32]              0       \n",
      "    Dnn-21          [[39, 221]]            [39, 32]              0       \n",
      "  Linear-103         [[39, 32]]            [39, 1]              33       \n",
      "===========================================================================\n",
      "Total params: 145,503\n",
      "Trainable params: 145,503\n",
      "Non-trainable params: 0\n",
      "---------------------------------------------------------------------------\n",
      "Input size (MB): 0.02\n",
      "Forward/backward pass size (MB): 0.15\n",
      "Params size (MB): 0.56\n",
      "Estimated Total Size (MB): 0.72\n",
      "---------------------------------------------------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'total_params': 145503, 'trainable_params': 145503}"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paddle.summary(model, (trn_x.shape[1],128))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "id": "f415b286-e88e-4650-b457-bd8d788e454d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:40:16.343328Z",
     "iopub.status.busy": "2022-03-06T17:40:16.341935Z",
     "iopub.status.idle": "2022-03-06T17:40:16.817233Z",
     "shell.execute_reply": "2022-03-06T17:40:16.816139Z",
     "shell.execute_reply.started": "2022-03-06T17:40:16.343252Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "id": "0683f3b3-516b-4d17-8347-931ba66ea3c3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:40:18.149166Z",
     "iopub.status.busy": "2022-03-06T17:40:18.148365Z",
     "iopub.status.idle": "2022-03-06T17:40:18.159353Z",
     "shell.execute_reply": "2022-03-06T17:40:18.157983Z",
     "shell.execute_reply.started": "2022-03-06T17:40:18.149091Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def auc(y_pred,y_true):\r\n",
    "    pred=y_pred\r\n",
    "    y=y_true\r\n",
    "    return roc_auc_score(y,pred)\r\n",
    "\r\n",
    "loss_func=nn.BCELoss()\r\n",
    "optimizer=paddle.optimizer.Adam(parameters=model.parameters(),learning_rate=0.001)\r\n",
    "metric_func=auc\r\n",
    "metric_name='auc'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "id": "dfa8b2d9-c967-416f-8844-ff72cd2d5eb8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:45:48.035376Z",
     "iopub.status.busy": "2022-03-06T17:45:48.034139Z",
     "iopub.status.idle": "2022-03-06T17:45:54.485679Z",
     "shell.execute_reply": "2022-03-06T17:45:54.484296Z",
     "shell.execute_reply.started": "2022-03-06T17:45:48.035302Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start training----\n",
      "========================================================2022-03-07  01:45:48\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=1, loss=0.000, auc = 1.000, val_loss=1.187, val_auc = 0.630\n",
      "\n",
      "================================================================================2022-03-07 01:45:48\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=2, loss=0.000, auc = 1.000, val_loss=1.198, val_auc = 0.628\n",
      "\n",
      "================================================================================2022-03-07 01:45:49\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=3, loss=0.000, auc = 1.000, val_loss=1.223, val_auc = 0.641\n",
      "\n",
      "================================================================================2022-03-07 01:45:49\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=4, loss=0.000, auc = 1.000, val_loss=1.240, val_auc = 0.640\n",
      "\n",
      "================================================================================2022-03-07 01:45:50\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=5, loss=0.000, auc = 1.000, val_loss=1.256, val_auc = 0.633\n",
      "\n",
      "================================================================================2022-03-07 01:45:51\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=6, loss=0.000, auc = 1.000, val_loss=1.276, val_auc = 0.640\n",
      "\n",
      "================================================================================2022-03-07 01:45:51\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=7, loss=0.000, auc = 1.000, val_loss=1.287, val_auc = 0.620\n",
      "\n",
      "================================================================================2022-03-07 01:45:52\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=8, loss=0.000, auc = 1.000, val_loss=1.305, val_auc = 0.616\n",
      "\n",
      "================================================================================2022-03-07 01:45:53\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=9, loss=0.000, auc = 1.000, val_loss=1.322, val_auc = 0.622\n",
      "\n",
      "================================================================================2022-03-07 01:45:53\n",
      "[step=10] loss:0.000,auc:1.000\n",
      "[step=20] loss:0.000,auc:1.000\n",
      "[step=30] loss:0.000,auc:1.000\n",
      "[step=40] loss:0.000,auc:1.000\n",
      "\n",
      "EPOCH=10, loss=0.000, auc = 1.000, val_loss=1.331, val_auc = 0.623\n",
      "\n",
      "================================================================================2022-03-07 01:45:54\n"
     ]
    }
   ],
   "source": [
    "#脚本风格\r\n",
    "import datetime\r\n",
    "\r\n",
    "epochs=10\r\n",
    "log_step_freq=10\r\n",
    "\r\n",
    "dfhistory=pd.DataFrame(columns=['epoch', 'loss', metric_name, 'val_loss', 'val_'+metric_name])\r\n",
    "\r\n",
    "print('start training----')\r\n",
    "nowtime=datetime.datetime.now().strftime('%Y-%m-%d  %H:%M:%S')\r\n",
    "print('======='*8+'%s'%nowtime)\r\n",
    "\r\n",
    "for epoch in range(1,epochs+1):\r\n",
    "    \r\n",
    "    #训练阶段\r\n",
    "    model.train()\r\n",
    "    loss_sum=0.0\r\n",
    "    metric_sum=0.0\r\n",
    "    step=1\r\n",
    "    \r\n",
    "    for step,(features,labels) in enumerate(train_loader,1):\r\n",
    "        #梯度清零\r\n",
    "        optimizer.clear_grad()\r\n",
    "        \r\n",
    "        #正向传播\r\n",
    "        predictions=model(features)\r\n",
    "        predictions=predictions.squeeze(1)\r\n",
    "        labels = paddle.cast(labels, dtype='float32')\r\n",
    "        loss=loss_func(predictions,labels)\r\n",
    "        try:\r\n",
    "            metric=metric_func(predictions,labels)\r\n",
    "            \r\n",
    "        except ValueError:\r\n",
    "            pass\r\n",
    "        \r\n",
    "        #反向传播\r\n",
    "        loss.backward()\r\n",
    "        optimizer.step()\r\n",
    "        \r\n",
    "        #打印出batch级日志\r\n",
    "        \r\n",
    "        loss_sum+=loss.item()\r\n",
    "        metric_sum+=metric.item()\r\n",
    "        \r\n",
    "        if step%log_step_freq==0:\r\n",
    "            print((\"[step=%d] loss:%.3f,\"+metric_name+\":%.3f\")%(step, loss_sum/step, metric_sum/step))\r\n",
    "\r\n",
    "    \r\n",
    "    #验证阶段\r\n",
    "    model.eval()\r\n",
    "    val_loss_sum=0.0\r\n",
    "    val_metric_sum=0.0\r\n",
    "    val_step=1\r\n",
    "    \r\n",
    "    for val_step,(features,labels) in enumerate(valid_loader,1):\r\n",
    "        \r\n",
    "            predictions=model(features)\r\n",
    "            predictions=predictions.squeeze(1)\r\n",
    "            labels = paddle.cast(labels, dtype='float32')\r\n",
    "            val_loss=loss_func(predictions,labels)\r\n",
    "            try:\r\n",
    "                val_metric=metric_func(predictions,labels)\r\n",
    "                \r\n",
    "            except ValueError:\r\n",
    "                pass\r\n",
    "            \r\n",
    "            val_loss_sum+=val_loss.item()\r\n",
    "            val_metric_sum+=val_metric.item()\r\n",
    "        \r\n",
    "    #记录日志\r\n",
    "    info = (epoch, loss_sum/step, metric_sum/step, val_loss_sum/val_step, val_metric_sum/val_step)\r\n",
    "    dfhistory.loc[epoch-1] = info\r\n",
    "    print((\"\\nEPOCH=%d, loss=%.3f, \" + metric_name + \" = %.3f, val_loss=%.3f, \" + \"val_\" + metric_name + \" = %.3f\") %info)\r\n",
    "    nowtime = datetime.datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\")\r\n",
    "    print('\\n' + '=========='* 8 + '%s' %nowtime)\r\n",
    "\r\n",
    "     "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b0845e2-c278-48d1-8733-9958b89519b1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T18:11:23.717966Z",
     "iopub.status.busy": "2022-03-06T18:11:23.716905Z",
     "iopub.status.idle": "2022-03-06T18:11:23.722515Z",
     "shell.execute_reply": "2022-03-06T18:11:23.721581Z",
     "shell.execute_reply.started": "2022-03-06T18:11:23.717890Z"
    }
   },
   "source": [
    "# 五、模型评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "a5241a48-4311-4618-9300-d6aa4c8823ca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:51:04.663877Z",
     "iopub.status.busy": "2022-03-06T17:51:04.663232Z",
     "iopub.status.idle": "2022-03-06T17:51:05.655469Z",
     "shell.execute_reply": "2022-03-06T17:51:05.653906Z",
     "shell.execute_reply.started": "2022-03-06T17:51:04.663807Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  if isinstance(obj, collections.Iterator):\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  return list(data) if isinstance(data, collections.MappingView) else data\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VdW5//HPAwlDAJFJVAIElRkUJIqtxbFWpCq9DqAiggPUsdaqVxzr5Sc//d222tpSKfWqt4ooxWK5lqp1oFZ/ikYLigOKFCGKEhARhTD53D/WPjsnyUlyEnJyAvm+X6/zOnvvtc7ez9k5Wc/ea0/m7oiIiAA0y3YAIiLSeCgpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUpF6ZWXMz+8rMetRn3Wwys4PMrN7P3Taz75rZyqTxZWY2Ip26dVjWvWZ2Q10/X818bzOzB+p7vpI9OdkOQLLLzL5KGs0DtgI7o/Efuvus2szP3XcCbeu7blPg7n3rYz5mdhFwrrsfkzTvi+pj3rLnU1Jo4tw9bpSjLdGL3P2ZquqbWY6772iI2ESk4an7SKoVdQ88amazzWwTcK6ZfcvMXjGzL8xsjZndbWa5Uf0cM3MzK4jGH4rK/2pmm8zsZTPrVdu6UflJZva+mW00s1+b2UtmNrGKuNOJ8YdmttzMNpjZ3UmfbW5md5nZejNbAYysZv3caGaPVJg23czujIYvMrN3o+/zYbQVX9W8is3smGg4z8wejGJ7GxhWoe5NZrYimu/bZnZqNH0w8BtgRNQ1ty5p3d6a9PmLo+++3sweN7P90lk3NTGzf4vi+cLMnjOzvkllN5jZJ2b2pZm9l/RdjzCzN6Lpn5nZz9JdnmSAu+ulF+4OsBL4boVptwHbgFMIGxGtgcOA4YQ9zQOA94HLo/o5gAMF0fhDwDqgEMgFHgUeqkPdfYBNwOio7CfAdmBiFd8lnRj/DLQHCoDPE98duBx4G8gHOgEvhH+VlMs5APgKaJM077VAYTR+SlTHgOOALcDBUdl3gZVJ8yoGjomGfw4sBDoAPYF3KtQdA+wX/U3OiWLoGpVdBCysEOdDwK3R8PeiGIcArYDfAs+ls25SfP/bgAei4f5RHMdFf6MbgGXR8EDgI2DfqG4v4IBo+DXg7Gi4HTA82/8LTfmlPQVJx4vu/j/u/o27b3H319x9kbvvcPcVwEzg6Go+P9fdi9x9OzCL0BjVtu7JwGJ3/3NUdhchgaSUZoy3u/tGd19JaIATyxoD3OXuxe6+HrijmuWsAJYSkhXACcAGdy+Kyv/H3Vd48BzwLJDyYHIFY4Db3H2Du39E2PpPXu4cd18T/U0eJiT0wjTmCzAOuNfdF7t7KTAFONrM8pPqVLVuqnMWMN/dn4v+RncQEstwYAchAQ2MuiD/Fa07CMm9t5l1cvdN7r4oze8hGaCkIOlYnTxiZv3M7C9m9qmZfQlMBTpX8/lPk4Y3U/3B5arq7p8ch7s7Ycs6pTRjTGtZhC3c6jwMnB0NnxONJ+I42cwWmdnnZvYFYSu9unWVsF91MZjZRDNbEnXTfAH0S3O+EL5fPD93/xLYAHRLqlObv1lV8/2G8Dfq5u7LgKsJf4e1UXfkvlHV84EBwDIze9XMRqX5PSQDlBQkHRVPx/wdYev4IHffC7iF0D2SSWsI3TkAmJlRvhGraFdiXAN0Txqv6ZTZOcB3zawbYY/h4SjG1sBc4HZC187ewNNpxvFpVTGY2QHAPcAlQKdovu8lzbem02c/IXRJJebXjtBN9XEacdVmvs0If7OPAdz9IXc/ktB11JywXnD3Ze5+FqGL8BfAY2bWahdjkTpSUpC6aAdsBL42s/7ADxtgmU8Ah5rZKWaWA1wJdMlQjHOAH5tZNzPrBFxXXWV3/xR4EXgAWObuH0RFLYEWQAmw08xOBo6vRQw3mNneFq7juDyprC2h4S8h5MdJhD2FhM+A/MSB9RRmAxea2cFm1pLQOP/D3avc86pFzKea2THRsq8lHAdaZGb9zezYaHlbotc3hC8w3sw6R3sWG6Pv9s0uxiJ1pKQgdXE1MIHwD/87wgHhjHL3z4CxwJ3AeuBA4J+E6yrqO8Z7CH3/bxEOgs5N4zMPEw4cx11H7v4FcBUwj3Cw9gxCckvHTwl7LCuBvwJ/SJrvm8CvgVejOn2B5H74vwEfAJ+ZWXI3UOLzTxK6ceZFn+9BOM6wS9z9bcI6v4eQsEYCp0bHF1oC/0k4DvQpYc/kxuijo4B3LZzd9nNgrLtv29V4pG4sdM2K7F7MrDmhu+IMd/9HtuMR2VNoT0F2G2Y2MupOaQncTDhr5dUshyWyR8lYUjCz+8xsrZktraHeYWa2w8zOyFQsssf4DrCC0DVxIvBv7l5V95GI1EHGuo/M7CjChSx/cPdBVdRpTuj/LAXuc/d0+m5FRCRDMran4O4vEA6uVecK4DHC1ZUiIpJlWbshXnRO978BxxJuSVBd3cnAZIA2bdoM69evX3XVRUSkgtdff32du1d3GjeQ3buk/hK4zt2/CdchVc3dZxJuU0BhYaEXFRU1QHgiInsOM6vpynwgu0mhEHgkSgidgVFmtsPdH89iTCIiTVrWkoK7J98S+QHgCSUEEZHsylhSMLPZwDFAZzMrJlyhmQvg7jMytVwREam7jCUFdz+75lpx3YmZikNERNKnK5pFRCSmpCAi0tjNmgUFBdCsWXifNStji8rm2UciIlKTWbNg8mTYvDmMf/RRGAcYt8s3t61EewoiIlXJ1BZ64vZCW7fCypXwzjvw+uvwj3/A00/D2ugmD//6F1xxRVlCSNi8GW68kUzQnoKISCqpttAvuABeeAEuvRQOOQTWrYNf/CLU2bKl7P2ii+Ckk+Ddd+HMM8uXbd4MDzwA55wDL78Mxx5bedmPPw6jR4dksWFD6vhWrcrI11ZSEJHGZ9assCW8ahX06AHTptW9q+Sbb2DjxtCAr18f3vfdFwoLYds2uPzysumJ90sugfvvr7yFvm0bzJwJPXuGpPD11yEp5OVB69Zl7198Eern5UHfvuXL8vKgf/9Q3r8/3Hdf5c8nyo8/HvLzoTjFQ/F61PSU2LrZ7R6yo9tciOzhKm6hQ2gsZ86Es84KDe7WrbD//qFs9uzQaCY36sOGwc03h/IOHcoa6YTzzw+NsXtoXPfaCzp3hk6dwvuoUXDaaWXdPMnMQpJp1y4z37+i6tZHLRKlmb3u7oU11dOegohk1+rV8MknUFISGvSrrkrdh37eeTB+fGiojz4aFi4MZbfeCu+/Dy1alDXsffqUffaGGyA3t6zB79QJuncPZWZh+an06BG6jFJNb6iEAGUNf33tOdVAewoiUmZXum22bw9b6iUlYUv6O98J0x99FF58sazRLymBVq1gUfRY6ZNOgiefTG8ZN98cGvaDDgpb8wBr1oRGuk2b0MjXl3raQm8stKcgsrupz370ui4/1amPJSXQu3f5Rr2kJDSOOTlwyy3w61+X76LJzQ1dPGbwt7/BY49Bly6hQS8oKNtSh7Alf8UVoaxLl7AXkGrrvWdPmDq18vT99qvX1RBr4C30xkJ7CiKNwa5ulW7dCl99Vf7Vv3/oK1++HJ59tnL5TTdBt26hwf75z8Mpkdu3V553+/Zhyz+hRYvQeL/5JnTsCH/8YziVsnPnsoa9c+fQuDdrFrp7arMFv4dtoTcW2lMQ2V2UlsK111Z9Lvqhh4Yt5ERj/vXX4X369HA64/z54fTFihYuDA3za6/BxReXTc/Lg7Ztw7Ru3cJW/V57pU4IAF9+Ca+8UtbYt2tXvpE/88zwqkptu3Sa6BZ6Y6E9BZH67rbZujVcfNSqVWhIN26E3/4WPvus/Ovf/x0mTIAlS2DIkNTzMoOionDWTdu25V/XXAOHHx72BObMqVw+fHg4qPr11yGGtm1Dv3vz5qmXVVCQ+sBqz57hAivZrWlPQSQd6d5CoLS0cqN+wAFw3HGhbOTIMO3TT8v61m+4ISSY7dvDcNu20LVrOEe+X7+w1Q1w4IEheZSUVI6vR4+wp/D++1V/h4MOCvOvSps24VWTadNSd9tMm1bzZ2WPoaQgTVNpabi69MYbU3fbTJ4cbjFw001hWpcuocsm2YQJISm0bBm2vgcNChcbde0aXodFjx5PbK3n5aWOpW1buOuu7DfI6rYRlBQkm3a12yb5AOayZeECps8/L3u1aQM/+lEonzQpnAKZKNuyBY46qupbBWzeXP72AnfcERr/ffcta/T32SeUmYUDuVUxqzohJDSWBnncOCWBJk7HFCQ7Up1h0qoVXH996Cf//POwdT1pUii7++5wo7BEo75hQ9jC/vDDUD5qFPz1r+WXMXAgLF0ahq+6KvSLd+xY9jrooHCAV/3o0gTomIJULZPnw2/ZEi5gSry+851wCuMzz8Bf/lI2/Zlnwn1kkpWWwk9/WjbevHm4sZhZuEBpzZrQmHfvHm5dkLjNAYSzc6ZMKWvwO3QI95BJuOuu1PFu25b9bhuRRkRJoalJ98Bq4iZibdqERn3lynAuevKNw9avhzvvDDfs+t3vwtb4li3ll/fRRyHxLFoE994b+tc7daqcEBLM4KWXyhr3hNtvD6+qFNa4AZRaY+m2EWkk1H3UVLiHRn7AgLDFXVHXruEsmERjv2ED7NxZdq777NnhVr8QLkjq2DE07nPnhgOsL74YbvebaPQTr+HDwxZ7xQuYdPqjSINS91FTs3VrON/944/DAdePPw6v886DE04IFx99+9tVf37t2tAHf/DB5Rv1Xr1C+ahR4bTITp1g771DYkj2ne+U3esmlYoXMOn0R5FGKWNJwczuA04G1rr7oBTl44DrAAM2AZe4+5JMxdMo1KUvf+fO0Le+bVu4sViisU9u9C+7LDTqw4eXfS43N/S5n3BCGO/TJ9zK4I47QvdPRT16VH8GTfv24VVf1G0j0ihlck/hAeA3wB+qKP8XcLS7bzCzk4CZwPAq6u7+UvXlT5oUzqS54oow7T/+I0xP3tI/55xwNWyzZjBxYujrb98+3J6gW7dwewIINwWbPz9My88PF0Ylb8136gRXXx1OqWwsW+g6/VGk0cnoMQUzKwCeSLWnUKFeB2Cpu3eraZ51OqZQn2fbuIezZDZtCq+dO8vu3f7UU2EZibIvvwxb61dfXXUfeuvWZQ10QUG4+jXRsHfrBsccA6efHso//DA06ulcnVqdbN+NU0QaXLrHFBpLUrgG6OfuF1VRPhmYDNCjR49hH6VqXKtS1fnwP/5xOB9+2zYYOzZMf/jh0Pf+5ZdlDXubNjBvXig//XT4859DIkjo3z88RxVgxIhwwDUhLy/0sz/1VNndIit/ubD1D+G9Yl+9iEg92G0ONJvZscCFQJVHKd19JqF7icLCwtplsVS3MSgtDX3rEK5STSSFZ54JZ9C0a1f2SnTPQDjY2rdvmJYoT1zVCiEBmYXpbduGe80nVPcUpwQlBBHJsqwmBTM7GLgXOMnd12dkIVXdxsAM3nij/GP17rsvvKpy4YXVL6u6B2nrbBsR2Q1kbdPUzHoAfwLGu3s1t4DcRVU11D16hNsVH3hgxhZdzrhx4SEhPXuGhNSzpx4aIiKNTiZPSZ0NHAN0NrNi4KdALoC7zwBuAToBv7VwDvuOdPq7aq0xbaHrbBsRaeQylhTc/ewayi8CUh5Yrlc6H15EJG1ZP9DcILSFLiKSFp3uIiIiMSUFERGJKSmIiEhMSUFERGJKCiIiElNSEBGRmJKCiIjElBRERCSmpCAiIjElBRERiSkpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISCxjScHM7jOztWa2tIpyM7O7zWy5mb1pZodmKhYREUlPJvcUHgBGVlN+EtA7ek0G7slgLCIikoaMJQV3fwH4vJoqo4E/ePAKsLeZ7ZepeEREpGbZPKbQDVidNF4cTavEzCabWZGZFZWUlDRIcCIiTdFucaDZ3We6e6G7F3bp0iXb4YiI7LGymRQ+BronjedH00REJEuymRTmA+dFZyEdAWx09zVZjEdEpMnLydSMzWw2cAzQ2cyKgZ8CuQDuPgNYAIwClgObgfMzFYuIiKQnY0nB3c+uodyByzK1fBERqb3d4kCziIg0DCUFERGJKSmIiEhMSUFERGJKCiIiElNSEBGRmJKCiIjElBRERCSmpCAiIjElBRERiSkpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISExJQUREYkoKIiISy2hSMLORZrbMzJab2ZQU5T3M7Hkz+6eZvWlmozIZj4iIVC9jScHMmgPTgZOAAcDZZjagQrWbgDnuPhQ4C/htpuIREZGaZXJP4XBgubuvcPdtwCPA6Ap1HNgrGm4PfJLBeEREpAaZTArdgNVJ48XRtGS3AueaWTGwALgi1YzMbLKZFZlZUUlJSSZiFRERsn+g+WzgAXfPB0YBD5pZpZjcfaa7F7p7YZcuXRo8SBGRpiKTSeFjoHvSeH40LdmFwBwAd38ZaAV0zmBMIiJSjZwMzvs1oLeZ9SIkg7OAcyrUWQUcDzxgZv0JSUH9QyJNzPbt2ykuLqa0tDTboez2WrVqRX5+Prm5uXX6fMaSgrvvMLPLgaeA5sB97v62mU0Fitx9PnA18Hszu4pw0Hmiu3umYhKRxqm4uJh27dpRUFCAmWU7nN2Wu7N+/XqKi4vp1atXneaRyT0F3H0B4QBy8rRbkobfAY7MZAwi0viVlpYqIdQDM6NTp07sygk52T7QLCICoIRQT3Z1PSopiIhITElBRHY7s2ZBQQE0axbeZ83atfl98cUX/Pa3tb+hwqhRo/jiiy9q/bmJEycyd+7cWn+uISgpiMhuZdYsmDwZPvoI3MP75Mm7lhiqSgo7duyo9nMLFixg7733rvuCGyElBRFpdI45pvIr0WZffz1s3ly+/ubNcOWVYXjdusqfrcmUKVP48MMPGTJkCIcddhgjRozg1FNPZcCAcLu2H/zgBwwbNoyBAwcyc+bM+HMFBQWsW7eOlStX0r9/fyZNmsTAgQP53ve+x5YtW9L6rs8++yxDhw5l8ODBXHDBBWzdujWOacCAARx88MFcc801APzxj39k0KBBHHLIIRx11FFpzb+2Mnr2kYhIfSsuTj19/fq6z/OOO+5g6dKlLF68mIULF/L973+fpUuXxqd13nfffXTs2JEtW7Zw2GGHcfrpp9OpU6dy8/jggw+YPXs2v//97xkzZgyPPfYY5557brXLLS0tZeLEiTz77LP06dOH8847j3vuuYfx48czb9483nvvPcws7qKaOnUqTz31FN26datTt1U6lBREpNFZuLDqsh49QpdRRT17hvfOnav/fDoOP/zwcuf533333cybNw+A1atX88EHH1RKCr169WLIkCEADBs2jJUrV9a4nGXLltGrVy/69OkDwIQJE5g+fTqXX345rVq14sILL+Tkk0/m5JNPBuDII49k4sSJjBkzhtNOO23XvmQV1H0kIruVadMgL6/8tLy8ML2+tGnTJh5euHAhzzzzDC+//DJLlixh6NChKa+8btmyZTzcvHnzGo9HVCcnJ4dXX32VM844gyeeeIKRI0cCMGPGDG677TZWr17NsGHDWL8ru0dVSCspmNmVZraXBf9lZm+Y2ffqPRoRkRqMGwczZ4Y9A7PwPnNmmF5X7dq1Y9OmTSnLNm7cSIcOHcjLy+O9997jlVdeqfuCKujbty8rV65k+fLlADz44IMcffTRfPXVV2zcuJFRo0Zx1113sWTJEgA+/PBDhg8fztSpU+nSpQurV6+ubvZ1km730QXu/iszOxHoAIwHHgServeIRERqMG7criWBijp16sSRRx7JoEGDaN26NV27do3LRo4cyYwZM+jfvz99+/bliCOOqLfltmrVivvvv58zzzyTHTt2cNhhh3HxxRfz+eefM3r0aEpLS3F37rzzTgCuvfZaPvjgA9yd448/nkMOOaTeYkmwdG41ZGZvuvvBZvYrYKG7zzOzf0ZPTGtQhYWFXlRU1NCLFZEMevfdd+nfv3+2w9hjpFqfZva6uxfW9Nl0jym8bmZPE5558JSZtQO+qXWkIiLSqKXbfXQhMARY4e6bzawjcH7mwhIR2f1ddtllvPTSS+WmXXnllZx/fuNtPtNNCt8CFrv712Z2LnAo8KvMhSUisvubPn16tkOotXS7j+4BNpvZIYRnIHwI/CFjUYmISFakmxR2RA+/GQ38xt2nA+0yF5aIiGRDut1Hm8zsesKpqCPMrBlQt2e9iYhIo5XunsJYYCvheoVPgXzgZxmLSkREsiKtpBAlgllAezM7GSh1dx1TEJHsqO8HKtRB27ZtqyxbuXIlgwYNasBo6k+6t7kYA7wKnAmMARaZ2RmZDExEJKVMPFBBYukeU7gROMzd1wKYWRfgGaBxPjpIRHZvqR6CMGYMXHpp9Q9UGDcuPFDhjArbrGncNnXKlCl0796dyy67DIBbb72VnJwcnn/+eTZs2MD27du57bbbGD16dK2+SmlpKZdccglFRUXk5ORw5513cuyxx/L2229z/vnns23bNr755hsee+wx9t9/f8aMGUNxcTE7d+7k5ptvZuzYsbVa3q5K95hCs0RCiKxP57NmNtLMlpnZcjObUkWdMWb2jpm9bWYPpxmPiDRVmXigAjB27FjmzJkTj8+ZM4cJEyYwb9483njjDZ5//nmuvvpq0rk1ULLp06djZrz11lvMnj2bCRMmUFpayowZM7jyyitZvHgxRUVF5Ofn8+STT7L//vuzZMkSli5dGt8dtSGlu6fwpJk9BcyOxscCC6r7gJk1B6YDJwDFwGtmNt/d30mq0xu4HjjS3TeY2T61/QIisgfKwgMVhg4dytq1a/nkk08oKSmhQ4cO7Lvvvlx11VW88MILNGvWjI8//pjPPvuMfffdN+35vvjii1xxxRUA9OvXj549e/L+++/zrW99i2nTplFcXMxpp51G7969GTx4MFdffTXXXXcdJ598MiNGjKj199hV6R5ovhaYCRwcvWa6+3U1fOxwYLm7r3D3bcAjhOsckk0Cprv7hmg5axERqU4GH6hw5plnMnfuXB599FHGjh3LrFmzKCkp4fXXX2fx4sV07do15bMU6uKcc85h/vz5tG7dmlGjRvHcc8/Rp08f3njjDQYPHsxNN93E1KlT62VZtZH2k9fc/THgsVrMuxuQfLPvYmB4hTp9AMzsJaA5cKu7P1lxRmY2GZgM0KNHj1qEICJ7nMQ9s2+8EVatCnsO06bVy720x44dy6RJk1i3bh1///vfmTNnDvvssw+5ubk8//zzfJRqD6UGI0aMYNasWRx33HG8//77rFq1ir59+7JixQoOOOAAfvSjH7Fq1SrefPNN+vXrR8eOHTn33HPZe++9uffee3f5O9VWtUnBzDYBqTrQDHB336selt8bOIZw7cMLZjbY3cs9fNTdZxL2VCgsLKxdh56I7Hnq+4EKkYEDB7Jp0ya6devGfvvtx7hx4zjllFMYPHgwhYWF9OvXr9bzvPTSS7nkkksYPHgwOTk5PPDAA7Rs2ZI5c+bw4IMPkpuby7777ssNN9zAa6+9xrXXXkuzZs3Izc3lnnvuqffvWJO0nqdQpxmbfYuw5X9iNH49gLvfnlRnBrDI3e+Pxp8Fprj7a1XNV89TENnz6HkK9ashnqdQF68Bvc2sl5m1AM4C5leo8zhhLwEz60zoTlqRwZhERKQaaR9TqC1332FmlwNPEY4X3Ofub5vZVKDI3edHZd8zs3eAncC17l7/T6IWEcmAt956i/Hjx5eb1rJlSxYtWpSliHZdxpICgLsvoMKpq+5+S9KwAz+JXiLShLk7ZpbtMGpl8ODBLF68ONthlLOrhwQy2X0kIpKWVq1asX79+l1u0Jo6d2f9+vW0atWqzvPI6J6CiEg68vPzKS4upqSkJNuh7PZatWpFfn5+nT+vpCAiWZebm0uvXr2yHYag7iMREUmipCAiIjElBRERiSkpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISExJQUREYkoKIiISU1IQEZGYkoKIiMSUFEREJJbRpGBmI81smZktN7Mp1dQ73czczAozGY+IiFQvY0nBzJoD04GTgAHA2WY2IEW9dsCVwKJMxSIiIunJ5J7C4cByd1/h7tuAR4DRKer9H+D/AaUZjEVERNKQyaTQDVidNF4cTYuZ2aFAd3f/SwbjEBGRNGXtQLOZNQPuBK5Oo+5kMysys6KSkpLMByci0kRlMil8DHRPGs+PpiW0AwYBC81sJXAEMD/VwWZ3n+nuhe5e2KVLlwyGLCLStGUyKbwG9DazXmbWAjgLmJ8odPeN7t7Z3QvcvQB4BTjV3YsyGJOIiFQjY0nB3XcAlwNPAe8Cc9z9bTObamanZmq5IiJSdzmZnLm7LwAWVJh2SxV1j8lkLCIiUjNd0SwiIjElBRERiSkpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISExJQUREYkoKIiISU1IQEZGYkoKIiMSUFEREJKakICIiMSUFERGJKSmIiEhMSUFERGJKCiIiEstoUjCzkWa2zMyWm9mUFOU/MbN3zOxNM3vWzHpmMh4REalexpKCmTUHpgMnAQOAs81sQIVq/wQK3f1gYC7wn5mKR0REapbJPYXDgeXuvsLdtwGPAKOTK7j78+6+ORp9BcjPYDwiIlKDTCaFbsDqpPHiaFpVLgT+mqrAzCabWZGZFZWUlNRjiCIikqxRHGg2s3OBQuBnqcrdfaa7F7p7YZcuXRo2OBGRJiQng/P+GOieNJ4fTSvHzL4L3Agc7e5bMxiPiIjUIJN7Cq8Bvc2sl5m1AM4C5idXMLOhwO+AU919bQZjERGRNGQsKbj7DuBy4CngXWCOu79tZlPN7NSo2s+AtsAfzWyxmc2vYnYiItIAMtl9hLsvABZUmHZL0vB3M7l8ERGpnUZxoFlERBoHJQUREYkpKYiISExJQUREYkoKIiISU1IQEZGYkoKIiMSUFEREJKakICIiMSV7bYmXAAAJ/klEQVQFERGJKSmIiEhMSUFERGJKCiIiElNSEBGRmJKCiIjElBRERCSmpCAiIjElBRERiSkpiIhITElBRERiSgoiIhLLaFIws5FmtszMlpvZlBTlLc3s0ah8kZkVZCKOWbOgoACaNQvvs2ZlYimKQ3Hs/nE0hhgUR5bjcPeMvIDmwIfAAUALYAkwoEKdS4EZ0fBZwKM1zXfYsGFeGw895J6X5w5lr7y8ML0hKQ7F0djjaAwxKI7MxQEUeRptt4W69c/MvgXc6u4nRuPXR0no9qQ6T0V1XjazHOBToItXE1RhYaEXFRWlHUdBAXz0UeXpzZtDfj60bAnLloVpP/kJzJtXvl7HjvD662H44ovh6afLl3fvDn//exgePx5eeql8eb9+sGBB1XHk5cHXX4fhkSPhgw/Klx91FNx/fxgeMQLWrClfftJJ8Otfh+FDD4UvvyxffsYZcMcdYbh/f1i+HHbsqBxHjx7QokXl6ZdcEtbL55/D8OGVy6+5Bn74Q1i9Go4/vnL5LbfAuefCu+/C6NFl0//1r9Rx5ORAr15l49OnwwknwAsvwEUXVa5///1w5JHw17/Cj39cufzRR2HIEJg7F268sXL5V1/BJ5+kjuOAA+D552H//WHGDPjlLyvXW7QI2reHX/wCfv/7yuVvvQW5uTB1Kjz8cPmyFi3gzTfDcPv2lf92AD17wsqVcPnl8Mwz5cvy88umXXABvPxy+fK+feHxx8PwWWfBkiXly4cOLYvplFPgySdT/0169gzrouJv78QTy9bJ8OGV4z/tNJg2LQwPHlx53uPHww03wNat4W+UUNVvNCcHevcuG7/qKpg0Kfz2Tjyxcv2bboJzzoH33guxVHT77eE3+cYb4Tda0fr1sHZt1XHcey98+9thvf3kJ5XrzZ4NhxwCf/pTiKWi+fPhoIPgwQdDLBU980z47XXqFP7/Kkr8NtJlZq+7e2FN9XLSn2WtdQNWJ40XAxWblbiOu+8ws41AJ2BdciUzmwxMBujRo0etgli1KvX0nTvhmGPCHzihX7/QCCdr165suH9/2Ly5fHmXLmXDgwaBWfny7t2rjyN5fkOGlJ9fYpkJhx4afqjJkv9JCgsrx3fAAWXDhx0W/kFSWb0azj678vT8/PCekwOHH165fL/9wnuLFmH5Fe2zT3jPyytfXjH5JezYUb7e3nuH9/btU89/r73Ce4cOYf1U1KZNeO/cOXX5o49WHcehh5Ylyq5dyzdcCc2bh/f99ktdnvg95OdXLs/NLRtOlRCg7HfTq1flzyfWLYS/c2LjIqFnz7LhAw8M25jJDjywbLhPH3jiiapjGDWq8m8zef4DB1ZefuK3AyEp7NxZvnz//cO7GRx8cNn0qn6jO3aE/7GERDwtWpSfntCxY3hv1Sp1eeK3lZeXunzu3OrjaNs2jLdvn/rzrVuXLSdVecuW4b1Tp9Tlid/ehg2p46iqTdll6exO1OUFnAHcmzQ+HvhNhTpLgfyk8Q+BztXNt7bdRz17lt/tSrx69qzVbHaZ4lAcjT2OxhCD4shcHKTZfZTJA80fA92TxvOjaSnrRN1H7YEK28K7Ztq0sCWQLC+vbLe2oSgOxdHY42gMMSiORhBHOpmjLi9C19QKoBdlB5oHVqhzGeUPNM+pab613VNwDwdkevZ0NwvvDX2gSHEojt0ljsYQg+LITBxk+0AzgJmNAn5JOBPpPnefZmZTo+Dmm1kr4EFgKPA5cJa7r6hunrU90CwiIo3jQDPuvgBYUGHaLUnDpcCZmYxBRETSpyuaRUQkpqQgIiIxJQUREYkpKYiISCyjZx9lgpmVACluGLFb6UyFq7abOK2P8rQ+ymhdlLcr66Onu3epqdJulxT2BGZWlM6pYU2F1kd5Wh9ltC7Ka4j1oe4jERGJKSmIiEhMSSE7ZmY7gEZG66M8rY8yWhflZXx96JiCiIjEtKcgIiIxJQUREYkpKTQgM+tuZs+b2Ttm9raZXZntmLLNzJqb2T/NrIrnfjUdZra3mc01s/fM7N3okbZNlpldFf2fLDWz2dFdlZsMM7vPzNaa2dKkaR3N7G9m9kH03qG+l6uk0LB2AFe7+wDgCOAyMxuQ5Ziy7Urg3WwH0Uj8CnjS3fsBh9CE14uZdQN+BBS6+yDC7ffPym5UDe4BYGSFaVOAZ929N/BsNF6vlBQakLuvcfc3ouFNhH/6btmNKnvMLB/4PnBvtmPJNjNrDxwF/BeAu29z9y+yG1XW5QCto6cy5gGfZDmeBuXuLxCeM5NsNPDf0fB/Az+o7+UqKWSJmRUQHi60KLuRZNUvgX8Hvsl2II1AL6AEuD/qTrvXzNpkO6hscfePgZ8Dq4A1wEZ3fzq7UTUKXd19TTT8KdC1vhegpJAFZtYWeAz4sbt/me14ssHMTgbWuvvr2Y6lkcgBDgXucfehwNdkoGtgdxH1lY8mJMv9gTZmdm52o2pcokds1vs1BUoKDczMcgkJYZa7/ynb8WTRkcCpZrYSeAQ4zsweym5IWVUMFLt7Ys9xLiFJNFXfBf7l7iXuvh34E/DtLMfUGHxmZvsBRO9r63sBSgoNyMyM0Gf8rrvfme14ssndr3f3fHcvIBxAfM7dm+yWoLt/Cqw2s77RpOOBd7IYUratAo4ws7zo/+Z4mvCB9yTzgQnR8ATgz/W9ACWFhnUkMJ6wVbw4eo3KdlDSaFwBzDKzN4EhwP/NcjxZE+0xzQXeAN4itFVN6pYXZjYbeBnoa2bFZnYhcAdwgpl9QNibuqPel6vbXIiISIL2FEREJKakICIiMSUFERGJKSmIiEhMSUFERGJKCiIRM9uZdKrwYjOrtyuKzawg+W6XIo1VTrYDEGlEtrj7kGwHIZJN2lMQqYGZrTSz/zSzt8zsVTM7KJpeYGbPmdmbZvasmfWIpnc1s3lmtiR6JW7P0NzMfh89I+BpM2sd1f9R9IyNN83skSx9TRFASUEkWesK3Udjk8o2uvtg4DeEu7sC/Br4b3c/GJgF3B1Nvxv4u7sfQrh/0dvR9N7AdHcfCHwBnB5NnwIMjeZzcaa+nEg6dEWzSMTMvnL3timmrwSOc/cV0Q0NP3X3Tma2DtjP3bdH09e4e2czKwHy3X1r0jwKgL9FD0fBzK4Dct39NjN7EvgKeBx43N2/yvBXFamS9hRE0uNVDNfG1qThnZQd0/s+MJ2wV/Fa9FAZkaxQUhBJz9ik95ej4f9P2SMixwH/iIafBS6B+BnU7auaqZk1A7q7+/PAdUB7oNLeikhD0RaJSJnWZrY4afxJd0+cltohunvpVuDsaNoVhCelXUt4atr50fQrgZnRXS13EhLEGlJrDjwUJQ4D7tZjOCWbdExBpAbRMYVCd1+X7VhEMk3dRyIiEtOegoiIxLSnICIiMSUFERGJKSmIiEhMSUFERGJKCiIiEvtfZk1VBqDF0qUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XucXfO9//HXO5PIpahIIiITmSAIVcEIrZ9D65Y6CHU/qQd+JdriqKNVWm39kJZznF60UdKWXsQl4iBV2rq2PS2aCQmCRKTBxCUjhJBEbp/fH981mT2TmVk7k9mzJ5n38/FYj73297sun71mz/rs73fdFBGYmZm1plu5AzAzs87PycLMzHI5WZiZWS4nCzMzy+VkYWZmuZwszMwsl5OFdRhJFZI+kLR9e05bTpJ2ktTu559LOlTS/IL3syUdWMy0bVjXLyR9s63zW9fQvdwBWOcl6YOCt32Aj4DV2ftzImLS+iwvIlYDm7f3tF1BROzSHsuRdBbwhYg4uGDZZ7XHsm3T5mRhLYqItTvr7JfrWRHxUEvTS+oeEas6IjYz61juhrI2k3SVpDsk3SZpCfAFSZ+S9ISkxZLekHSdpB7Z9N0lhaSq7P0tWf0DkpZIelzSsPWdNqv/nKQ5kt6T9BNJf5N0RgtxFxPjOZLmSnpX0nUF81ZI+qGkRZLmAaNb2T7fknR7k7IJkn6QjZ8l6YXs87yc/epvaVm1kg7OxvtI+m0W2yxgnybTXiZpXrbcWZKOycr3AH4KHJh18b1dsG0vL5j/S9lnXyTpHkmDitk267md1+m+k/S/hX+zbD0vZp/jOUl7trQu6wAR4cFD7gDMBw5tUnYVsAI4mvTDozewL7AfqdW6AzAHOC+bvjsQQFX2/hbgbaAa6AHcAdzShmm3AZYAY7K6/wBWAme08FmKifFe4ONAFfBO/WcHzgNmAZVAP+Av6d+o2fXsAHwAfKxg2QuB6uz90dk0Aj4LLAM+mdUdCswvWFYtcHA2fi3wGNAXGAo832Tak4BB2d/k37IYBmZ1ZwGPNYnzFuDybPzwLMaRQC/geuCRYrbNem7nnZpuN+B/6/9mwKnAa6REKGBnYEi5/w+68uCWhW2o/42I30XEmohYFhHTIuLJiFgVEfOAicBBrcw/JSJqImIlMIm0k1rfaY8CZkTEvVndD0mJpVlFxvj9iHgvIuaTdsz16zoJ+GFE1EbEIuDqVtYzD3iOlMQADgPejYiarP53ETEvkkeAh4FmD2I3cRJwVUS8GxGvkFoLheudHBFvZH+TW0mJvrqI5QKMBX4RETMiYjlwCXCQpMqCaVraNo204btQ6Czg6oiYnm2fORHxWpHzWgk4WdiGavQPLGlXSb+X9Kak94ErgP6tzP9mwfhSWj+o3dK02xXGERFB+iXerCJjLGpdwCutxAtwK+lXMqRf+bcWxHGUpCclvSNpMelXfWvbqt6g1mKQdIakmVn3z2Jg1yKXC+nzrV1eRLwPvAsMLpimqL9ZG74LhYYALxc5rXUAJwvbUE1PG72R9Gt6p4jYEvgOqRuhlN4gdQsBIEk03rk1tSExvkHakdXLO7V3MnCopMGkFsatWYy9gSnA90ldRFsBfyoyjjdbikHSDsDPgC8D/bLlvliw3LzTfF8ndW3VL28LUnfXgiLiaqq17fxhtvw+BdNvWzD+GrBjG9ZpJeJkYe1tC+A94ENJI4BzOmCd9wF7SzpaUnfgAmBAiWKcDHxV0mBJ/YBvtDZxRLxJ6ov/FTA7Il7KqnoCmwF1wGpJRwGHrEcM35S0ldJ1KOcV1G1OSgh1pLx5NqllUe8toLL+QHMzbgO+KOmTknqSktlfI6LFllorWtvOb2bDF7KTBsZRkKSAXwAXS9pLyXBJhQnSOpiThbW3i4DTSQecbyQdiC6piHgLOBn4AbCI9Iv0adJ1Ie0d489IxxaeBaaRWgd5biUdsF7bBRURi4ELgbtJB4lPICW9YnyX1MKZDzwA/KZguc8APwH+kU2zC/BkwbwPAi8Bb0kq7E6qn/8PpO6iu7P5tycdx2iLFrdz1lV4NvBN0vGlnQrjjIjbgGuyed4H/ofUwrEyUfqbmW06JFWQulNOiIi/ljses02BWxa2SZA0OuuW6Ql8m3Tq7D/KHJbZJsPJwjYV/weYR+qrPwI4LiJa6oYys/XkbigzM8vlloWZmeXaZG4k2L9//6iqqip3GGZmG5Xp06e/HRGtnWoObELJoqqqipqamnKHYWa2UZGUdxcCwN1QZmZWBCcLMzPL5WRhZma5nCzMzCyXk4WZmeUqWbKQdJOkhZKea6Fe2WMW50p6RtLeBXWnS3opG04vVYwAkyZBVRV065ZeJ00q5do6bwyOw3FsDHF0hhi6bBylegQf8C/A3sBzLdQfSbpjpoD9gSez8q1Jt23YmnSXyXlA37z17bPPPrG+brklok+fCGgY+vRJ5R2lM8TgOBzHxhBHZ4hhU4wDqIki9uklvd2HpCrgvoj4RDN1N5KeBXxb9n42cHD9EBHnNDddS6qrq2N9r7OoqoJXmjnDuGdP2H9/OOkk+MpXYOlSOPLIdac744w0vP02nHDCuvVf/jKcfDK89hqcdtq69RddBOef33IM990Hhx4KM2bAV7+67jTf+x58+tPw97/DN7+5bv2PfgQjR8JDD8FVV61bf+ONsMsu8LvfwYknwkfN3Elp6FC45hr42c/WrZsyBfr3h1/9Kg1N3X8/9OkD118PkyevW//YY+n12mvTZwV44onm46j/m9Tr1w/uuiuNX3opPP544+krK+GWW9L4V7+atmGhnXeGiRPT+LhxMGdO4/qnnoIlS1qO41Ofgu9/P5UdfzwsWtR4ukMOgW9/O41/7nOwbFnj+qOOgq99LY0ffPC666n/7m2/ffr+NNWvX/rebch37+ijYfZsOKeZp3lcdlnj715Lf5eBA2HXXdctX5/v3n//97r1v/0tDBkCd9zR8N0r9rsBbfvu1evdGx54II1feSU8/HDj+n/8Y92/Z2EcG/rdGzkybT+AL3wBaps8SaT+u9fS/mvoUJg/f93ylkiaHhG5j90t5zGLwTR+NGRtVtZS+TokjZNUI6mmrq5uvQN49dXmy5v7QpZKZ4ihtfW1FF9Hx9HR26O5RFGOOJruKOo1TU6l1tLnXriw/DF09N+kuURRjjha+t8s2f9sMc2Ptg5AFS13Q90H/J+C9w+THir/NeCygvJvA1/LW1dbuqGGDm3chKsfhg5d70W1WWeIwXE4jo0hjs4Qw6YYB0V2Q5WzZbGAxs8RrszKWipvd+PHp6ZqoT59UnlH6QwxOA7HsTHE0Rli6NJxFJNR2jrQesviX2l8gPsfWfnWwD9JB7f7ZuNb562rLS2LiHQwaOjQCCm9dvRBqs4Sg+NwHBtDHJ0hhk0tDsp9gFvSbaSD1f1JD4n/LtAjS1A3SBLwU2A0sBQ4MyJqsnn/L+nZvADjI+LmvPW15QC3mVlXV+wB7pLddTYiTs2pD+DcFupuAm4qRVxmZrb+fAW3mZnlcrIwM7NcThZmZpbLycLMzHI5WZiZWS4nCzMzy+VkYWZmuZwszMwsl5OFmZnlcrIwM7NcThZmZpbLycLMzHI5WZiZWS4nCzMzy+VkYWZmuZwszMwsl5OFmZnlKmmykDRa0mxJcyVd0kz9UEkPS3pG0mOSKgvqVkuakQ1TSxmnmZm1rmSPVZVUAUwADgNqgWmSpkbE8wWTXQv8JiJ+LemzwPeB07K6ZRExslTxmZlZ8UrZshgFzI2IeRGxArgdGNNkmt2AR7LxR5upNzOzTqCUyWIw8FrB+9qsrNBM4PPZ+HHAFpL6Ze97SaqR9ISkY5tbgaRx2TQ1dXV17Rm7mZkVKPcB7q8BB0l6GjgIWACszuqGRkQ18G/AjyTt2HTmiJgYEdURUT1gwIAOC9rMrKsp2TEL0o5/SMH7yqxsrYh4naxlIWlz4PiIWJzVLche50l6DNgLeLmE8ZqZWQtK2bKYBgyXNEzSZsApQKOzmiT1l1Qfw6XATVl5X0k966cBDgAKD4ybmVkHKlmyiIhVwHnAH4EXgMkRMUvSFZKOySY7GJgtaQ4wEBiflY8AaiTNJB34vrrJWVRmZtaBFBHljqFdVFdXR01NTbnDMDPbqEianh0fblW5D3CbmdlGwMnCzMxyOVmYmVkuJwszM8vlZGFmZrmcLMzMLJeThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVmYmVkuJwszM8vlZGFmZrmcLMzMLJeThZmZ5XKyMDOzXE4WZmaWq6TJQtJoSbMlzZV0STP1QyU9LOkZSY9JqiyoO13SS9lweinjNDOz1pUsWUiqACYAnwN2A06VtFuTya4FfhMRnwSuAL6fzbs18F1gP2AU8F1JfUsVq5mZta6ULYtRwNyImBcRK4DbgTFNptkNeCQbf7Sg/gjgwYh4JyLeBR4ERpcwVjMza0Upk8Vg4LWC97VZWaGZwOez8eOALST1K3JeJI2TVCOppq6urt0CNzOzxsp9gPtrwEGSngYOAhYAq4udOSImRkR1RFQPGDCgVDGamXV53Uu47AXAkIL3lVnZWhHxOlnLQtLmwPERsVjSAuDgJvM+VsJYzcysFaVsWUwDhksaJmkz4BRgauEEkvpLqo/hUuCmbPyPwOGS+mYHtg/PyszMrAxKliwiYhVwHmkn/wIwOSJmSbpC0jHZZAcDsyXNAQYC47N53wGuJCWcacAVWZmZmZWBIqLcMbSL6urqqKmpKXcYZmYbFUnTI6I6b7pyH+A2M7ONgJOFmZnlcrIwM7NcThZmZpbLycLMzHI5WZiZWS4nCzMzy+VkYWZmuZwszMwsl5OFmZnlcrIwM7NcThZmZpbLycLMzHI5WZiZWS4nCzMzy+VkYWZmuUqaLCSNljRb0lxJlzRTv72kRyU9LekZSUdm5VWSlkmakQ03lDJOMzNrXfdSLVhSBTABOAyoBaZJmhoRzxdMdhnpcas/k7QbcD9QldW9HBEjSxWfmZkVr5Qti1HA3IiYFxErgNuBMU2mCWDLbPzjwOsljMfMzNqolMliMPBawfvarKzQ5cAXJNWSWhXnF9QNy7qn/izpwBLGaWZmOcp9gPtU4FcRUQkcCfxWUjfgDWD7iNgL+A/gVklbNp1Z0jhJNZJq6urqOjRwM7OupJTJYgEwpOB9ZVZW6IvAZICIeBzoBfSPiI8iYlFWPh14Gdi56QoiYmJEVEdE9YABA0rwEczMDEqbLKYBwyUNk7QZcAowtck0rwKHAEgaQUoWdZIGZAfIkbQDMByYV8JYzcysFSU7GyoiVkk6D/gjUAHcFBGzJF0B1ETEVOAi4OeSLiQd7D4jIkLSvwBXSFoJrAG+FBHvlCpWMzNrnSKi3DG0i+rq6qipqSl3GGZWAitXrqS2tpbly5eXO5SNVq9evaisrKRHjx6NyiVNj4jqvPlL1rIwM2svtbW1bLHFFlRVVSGp3OFsdCKCRYsWUVtby7Bhw9q0jHKfDWVmlmv58uX069fPiaKNJNGvX78Napk5WZjZRsGJYsNs6PZzsjAzs1xOFma2yZk0CaqqoFu39Dpp0oYtb/HixVx//fXrPd+RRx7J4sWLN2zlnYSThZltUiZNgnHj4JVXICK9jhu3YQmjpWSxatWqVue7//772Wqrrdq+4k7EycLMNjoHH7zuUL8vv/RSWLq08fRLl8IFF6Txt99ed948l1xyCS+//DIjR45k33335cADD+SYY45ht912A+DYY49ln332Yffdd2fixIlr56uqquLtt99m/vz5jBgxgrPPPpvdd9+dww8/nGXLlrW4vp///Ofsu+++7Lnnnhx//PEszT7QGWecwZQpU9ZOt/nmm68dv+aaa9hjjz3Yc889ueSSdZ4IscGcLMxsk1Jb23z5okVtX+bVV1/NjjvuyIwZM/iv//ovnnrqKX784x8zZ84cAG666SamT59OTU0N1113HYuaWdlLL73Eueeey6xZs9hqq6246667Wlzf5z//eaZNm8bMmTMZMWIEv/zlL1uN74EHHuDee+/lySefZObMmVx88cVt/7At8HUWZrbReeyxluu23z51PTU1dGh67d+/9fmLMWrUqEbXK1x33XXcfffdALz22mu89NJL9OvXr9E8w4YNY+TI9IieffbZh/nz57e4/Oeee47LLruMxYsX88EHH3DEEUe0Gs9DDz3EmWeeSZ8+fQDYeuut2/KxWuWWhZltUsaPh2yfuVafPqm8vXzsYx9bO/7YY4/x0EMP8fjjjzNz5kz22muvZq9n6Nmz59rxioqKVo93nHHGGfz0pz/l2Wef5bvf/e7a5XXv3p01a9YAsGbNGlasWNFeHylXUclC0v6Stih4v6Wk/UoXlplZ24wdCxMnppaElF4nTkzlbbXFFluwZMmSZuvee+89+vbtS58+fXjxxRd54okn2r6izJIlSxg0aBArV65kUsGR+aqqKqZPnw7A1KlTWblyJQCHHXYYN99889pjG++80/630iu2G+pnwN4F7z9opszMrFMYO3bDkkNT/fr144ADDuATn/gEvXv3ZuDAgWvrRo8ezQ033MCIESPYZZdd2H///Td4fVdeeSX77bcfAwYMYL/99lubqM4++2zGjBnDnnvuyejRo9e2cEaPHs2MGTOorq5ms80248gjj+R73/veBsdRqKgbCUqa0fR52JKeiYhPtms0G8A3EjTbdL3wwguMGDGi3GFs9JrbjsXeSLDYYxbzJP27pB7ZcAF+voSZWZdRbLL4EvBp0pPuaoH9gHGlCsrMrCs499xzGTlyZKPh5ptvLndYzSrqmEVELCQ96c7MzNrJhAkTyh1C0YpKFpJuJj3JrpGI+L/tHpGZmXU6xXZD3Qf8PhseBrYknRHVKkmjJc2WNFfSOtefS9pe0qOSnpb0jKQjC+ouzeabLan1K1LMzKykiu2GanRduqTbgP9tbR5JFcAE4DDScY5pkqZGxPMFk10GTI6In0naDbgfqMrGTwF2B7YDHpK0c0SsLvJzmZlZO2rrFdzDgW1yphkFzI2IeRGxArgdGNNkmiC1UgA+DryejY8Bbo+IjyLin8DcbHlmZlYGxV7BvUTS+9nwHvA7IO9OVYOB1wre12ZlhS4HviCpltSqOH895kXSOEk1kmrq6uqK+Shm1hW09wMt1lPh3WA3FcV2Q20haWtSi6JXfXE7rP9U4FcR8d+SPgX8VtInip05IiYCEyFdlNcO8ZjZxq7+gRb19ymvf6AFtO9l3V1MsWdDnQVcAFQCM4D9gceBz7Yy2wJgSMH7yqys0BeB0QAR8bikXkD/Iuc1s66quYdQnHQSfOUrrT/QYuzY9ECLE05oXJ9zG9pLLrmEIUOGcO655wJw+eWX0717dx599FHeffddVq5cyVVXXcWYMU172tf1wQcfMGbMmHXmmz9/PkcddRTPPfccANdeey0ffPABl19+OXPnzuVLX/oSdXV1VFRUcOedd7Ljjjvmrqs9FXvM4gJgX+CViPgMsBeQ96zAacBwScMkbUY6YD21yTSvAocASBpBarXUZdOdIqmnpGGkFs0/iozVzLqyEjzQ4uSTT2by5Mlr30+ePJnTTz+du+++m6eeeopHH32Uiy66iGJun9SrV6/1nm/s2LGce+65zJw5k7///e8MGjSozZ+lrYq9keDyiFguCUk9I+JFSbu0NkNErJJ0HvBHoAK4KSJmSboCqImIqcBFwM8lXUjq1joj0labJWky8DywCjjXZ0KZ2Vod/ECLvfbai4ULF/L6669TV1dH37592Xbbbbnwwgv5y1/+Qrdu3ViwYAFvvfUW2267bavLigi++c1vrjNfS5YsWcKCBQs47rjjgJRsyqHYZFEraSvgHuBBSe8Czfw1GouI+0kHrgvLvlMw/jxwQAvzjgfa8Q70ZtYljB/f+JgFtMsDLU488USmTJnCm2++ycknn8ykSZOoq6tj+vTp9OjRg6qqqmafY9FUS/MVPqsCKGpZHamobqiIOC4iFkfE5cC3gV8Cx5YyMDOzNinFAy1IXVG33347U6ZM4cQTT+S9995jm222oUePHjz66KO80lxrphktzTdw4EAWLlzIokWL+Oijj7jvvvuA9CyNyspK7rnnHgA++uijtc+t6Ejr/VjViPhzKQIxM2s37f1AC2D33XdnyZIlDB48mEGDBjF27FiOPvpo9thjD6qrq9l1112LDK35+Xr06MF3vvMdRo0axeDBgxst77e//S3nnHMO3/nOd+jRowd33nknO+ywQ7t+vjxFPc9iY+DnWZhtuvw8i/bREc+zMDOzLmy9u6HMzCzfs88+y2mnndaorGfPnjz55JNlimjDOFmY2UYhIpBU7jCKtsceezBjxoxyh7HWhh5ycDeUmXV6vXr1YtGiRRu8w+uqIoJFixZt0DUablmYWadXWVlJbW0tvmFo2/Xq1YvKyso2z+9kYWadXo8ePRg2bFi5w+jS3A1lZma5nCzMzCyXk4WZmeVysjAzs1xOFmZmlsvJwszMcjlZmJlZLicLMzPLVdJkIWm0pNmS5kq6pJn6H0qakQ1zJC0uqFtdUNf02d1mZtaBSnYFt6QKYAJwGFALTJM0NXuUKgARcWHB9OcDexUsYllEjCxVfGZmVrxStixGAXMjYl5ErABuB8a0Mv2pwG0ljMfMzNqolMliMPBawfvarGwdkoYCw4BHCop7SaqR9ISkZp/3LWlcNk2NbzBmZlY6neUA9ynAlIhYXVA2NHvU378BP5K0Y9OZImJiRFRHRPWAAQM6KlYzsy6nlMliATCk4H1lVtacU2jSBRURC7LXecBjND6eYWZmHaiUyWIaMFzSMEmbkRLCOmc1SdoV6As8XlDWV1LPbLw/cADwfNN5zcysY5TsbKiIWCXpPOCPQAVwU0TMknQFUBMR9YnjFOD2aPwIrBHAjZLWkBLa1YVnUZmZWcfSpvKYwurq6qipqSl3GGZmGxVJ07Pjw63qLAe4zcysE3OyMDOzXE4WZmaWy8nCzMxyOVmYmVkuJwszM8vlZGFmZrmcLMzMLJeThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVmYmVkuJwszM8vlZGFmZrmcLMzMLJeThZmZ5SppspA0WtJsSXMlXdJM/Q8lzciGOZIWF9SdLumlbDi9lHGamVnrSvYMbkkVwATgMKAWmCZpauGztCPiwoLpzwf2ysa3Br4LVAMBTM/mfbdU8ZqZWctK2bIYBcyNiHkRsQK4HRjTyvSnArdl40cAD0bEO1mCeBAYXcJYzcysFaVMFoOB1wre12Zl65A0FBgGPLI+80oaJ6lGUk1dXV27BG1mZuvqLAe4TwGmRMTq9ZkpIiZGRHVEVA8YMKBEoZmZWSmTxQJgSMH7yqysOafQ0AW1vvOamVmJlTJZTAOGSxomaTNSQpjadCJJuwJ9gccLiv8IHC6pr6S+wOFZmZmZlUHJzoaKiFWSziPt5CuAmyJilqQrgJqIqE8cpwC3R0QUzPuOpCtJCQfgioh4p1SxmplZ61Swj96oVVdXR01NTbnDMDPbqEiaHhHVedN1lgPcZmbWiTlZmJlZLicLMzPL5WRhZma5nCzMzCyXk4WZmeVysjAzs1xOFmZmlsvJwszMcjlZmJlZLicLMzPL5WRhZma5nCzMzCyXk4WZmeVysjAzs1xOFmZmlqukyULSaEmzJc2VdEkL05wk6XlJsyTdWlC+WtKMbFjncaxmZtZxSvZYVUkVwATgMKAWmCZpakQ8XzDNcOBS4ICIeFfSNgWLWBYRI0sVn5mZFa+ULYtRwNyImBcRK4DbgTFNpjkbmBAR7wJExMISxmNmZm1UymQxGHit4H1tVlZoZ2BnSX+T9ISk0QV1vSTVZOXHljBOMzPLUbJuqPVY/3DgYKAS+IukPSJiMTA0IhZI2gF4RNKzEfFy4cySxgHjALbffvuOjdzMrAspZctiATCk4H1lVlaoFpgaESsj4p/AHFLyICIWZK/zgMeAvZquICImRkR1RFQPGDCg/T9BVzNpElRVQbdu6XXSpHJHZGadRCmTxTRguKRhkjYDTgGantV0D6lVgaT+pG6peZL6SupZUH4A8DxWOpMmwbhx8MorEJFex40rT8Jw0mrM28M6gZJ1Q0XEKknnAX8EKoCbImKWpCuAmoiYmtUdLul5YDXw9YhYJOnTwI2S1pAS2tWFZ1FZCXzrW7B0aeOypUvhG9+ArbeGioq0s+rWLY3vuy/06QOvvw61tY3ru3WDXXeFHj1g0SJYvLhxXbduMGhQev3wQ1ixoqF8yhQ477yGWOqTFsDYsR27TTqDSZPg7LNh2bL0vqtvDysbRUS5Y2gX1dXVUVNTU+4wNj7PPAPXXQe//OX6zffSS7DTTnDNNXBJM5fQvPkmDBwI3/42XHXVuvUffpiSzVe/Cj/+cf76eveGM85ISWbIkDRev5zevVOi2disWpW204IFKenWv551FuywQ0qcJ52UWnpNDR0K8+d3eMi26ZE0PSKq86Yr9wFu6wh1dfD002mYMSO9XnMNjBmTfvXfcw/06gXLl687b2Ul3HknrFkDq1en1zVrYHB2YtsJJ8AnPtFQXj9stVWqP/542Hnndes32yzVH3ccDBvWsOyvf735z7BsGUyenFoq223XkCxOPRUeeAC23TYlku22S/HUJ6hp01KrZ7vtYMCANF6sSZNSi+vVV2H77WH8+PX7Nb9kCTz+eEoChQnhP/4DDjoIHn4YRo9uPE/37qluhx3S0NKPuVdfhTvugIUL09/RJ3hYibllsSmJSL82Z8xIO/NRo+Dll1MLoN7228Nee8EFF8BnPpN20BLcemvq3ijsiurTByZO7Njujqqq1NXSVP0v6Y8+gnfeSYkBUiKbMSPtiN94I71utx384Q+pfu+9U3KElCgGDkw76PqW1A03pPL6RDNoEGyzDdx+e/PbY8KElAC32CIlg1/8onEiWLAgdd2dfTbMmpUSV71+/dI6xo+Ho49OrYp7701/q+22S68DBjRuJbW2PT71qRQnpL/pmDEp+X7yk23Y8NZVFduyICI2iWGfffaJNrnlloihQyOk9HrLLW1bTrmsWRNx0UURBx8csdVWESllRJx9dqpfvTriBz+IePjhiEUubAAFAAALUklEQVSLWl9WZ9gWt9wS0adPw+eA9L6tsdTURNx1V8RPfxrxrW9FnHlmxJVXNtRvs03jdUHESSelz9+0vH64+OI07+LF6X2vXhE77hhx4IERp5wS8fvfp/plyyL++teIl19O46XYHi++GHHNNRGf/nT6ux1xRMO806ZFrFzZtvVal0E6hpy7jy37Tr69hjYli/beMbVV3k76ww8jHn884vrrUxKoro444YSG+j32iBg1KuKccyJuuCHiyScjli7tyE/QvjoyaX30UcSrr0Y88UTE3XdHTJiQdvZSy8niz39O865ZkxLwmjWliy+i+O3x5psRzz/fMC5FbL11xGmnpYT5wQeljdM2SsUmi67dDdVSE79799R98Otfpyb9ww/D9denfv36oXdv+NrXUl/5jBnwt781lNdP85nPpPcLF6a+9sK63r1Tv31z3T89e8Lpp8ONN6b3Bx0Ef/lLGu/bN3U5HHooXHppKotIXUnWfvK6wzq75cvh/vtTN9fvfgfvvpu+d3fcAcccU+7orBNxN1QxWvv1ePTREXPmpOnuuiv9et9pp4jKyoh+/SI+9rGI2bNT/bXXNr+M2tpUf/nlzdcvXtx6d8eKFWn+3/8+/eqdP7/0v2It6SytzvawcmXEo49GXHBBxCuvpLLf/CZ1Xf3nfzZ8j61Lwi2LIrTXr8fly+H999Pr8uXpzJ3ly1MLYLPN0oHOZ59tXLd8eWqZ9OzZ/BkvUjr4bOWzoWdDdWZ33glXXw1PPZXejxiRDpBfddX6nTHWVXWW70Y7xOGWRTE6w6/HlloWQ4d2XAzWdc2fH3HddRGHHBKx994N5TfcEHH//RHLlzeUdYYTIDqDjthvrF6dehaWLUvHmgpPkFi4MOKNNyJ+8pOI3r03OA58gLtI5f4H6AwJyyyi4cypVasitt02fRc33zzixBMjvvzldtkxbVRWrkw75Zkz05ll9fr2bf0H3rHHRuy+e8Suu0YMHx6xww4Rp5/eMP/ee0f075+Ws+WWqUt77NjWl3/mmQ31FRXNr7+NPzSLTRa+KG/s2PJ2LdSvuzM0aa1r657tDioq4J//hEceSQfI770X3npr3emXLk23Znn/fdhySzjwwPT9Xbo0XUOyxRapvGfP9ouxPbp/amtT9/PChelzLVyYLgr9f/8v1X/xi+kzL1rUMM8uu8CLL6bxd99tfrmvvppehw5tuP1NRUUadtutYbojjkgXw9bXVVTAnns21F98cbqeqLB+jz0a6n/yk/T6la+0Hkc769rHLMws35o1KZHk7SsmT4YTT4SHHoLDDmso32yzlDTuuAM++1n4+9/he99LZYXDGWekOwbU1qYdc9P6e+9t/kLJa66BAw5IO9xu3eBPf0oxLFzYkBAWL4a5c9OxwDPPhF/9qnHsQ4Y07GR//GOYPTtdwLnNNmkYPBj23z/Vd5Yz5dopDt/uw8zaR7du6Vd8czumIUPgySfT1ewDB6ay3XZLp52//34alixJr9ttl+o//DBd8f7iiw11y5fD5z6XksUDDzTcLLHQoEHN3+zy/PPTeF0d9O+fTjO/7rqGnf2226ZT4FesSK2c88+HU05pqO/fv+H2M5DubtCa8eObT1rjx7c+X3vr4DjcsjCzfPW3sC/V7WBWrGjoclm4EObMaUg29cPFF7fcurn77tS907t3ukFjRUVprz3qgmdDOVmYWXHKvYPsLN0/m5hik8VGeF9nMyuLsWPTTnnNmvTa0b+kx49PrZlC5ej+6aKcLMxs4zB2bOr2Gjo0dTENHdrxd0XuwnyA28w2HuU+1b0LK2nLQtJoSbMlzZXUzOPUQNJJkp6XNEvSrQXlp0t6KRtOL2WcZmbWupK1LCRVABOAw4BaYJqkqVHwLG1Jw4FLgQMi4l1J22TlWwPfBaqBAKZn87ZwNYyZmZVSKVsWo4C5ETEvIlYAtwNjmkxzNjChPglExMKs/AjgwYh4J6t7EGjy/EkzM+sopUwWg4HXCt7XZmWFdgZ2lvQ3SU9IGr0e8yJpnKQaSTV1dXXtGLqZmRUq99lQ3YHhwMHAqcDPJW1V7MwRMTEiqiOiesCAASUK0czMSnk21AJgSMH7yqysUC3wZESsBP4paQ4peSwgJZDCeR9rbWXTp09/W1IzV+xsVPoDb5c7iE7E26Mxb48G3haNbcj2GFrMRCW7gltSd2AOcAhp5z8N+LeImFUwzWjg1Ig4XVJ/4GlgJNlBbWDvbNKngH0i4p2SBNtJSKop5krKrsLbozFvjwbeFo11xPYoWcsiIlZJOg/4I1AB3BQRsyRdQbp/+tSs7nBJzwOrga9HxCIASVeSEgzAFZt6ojAz68w2mXtDbQr8a6kxb4/GvD0aeFs01hHbo9wHuK2xieUOoJPx9mjM26OBt0VjJd8eblmYmVkutyzMzCyXk4WZmeVysugEJA2R9GjBDRVznuu46ZNUIelpSfeVO5Zyk7SVpCmSXpT0gqRPlTumcpJ0YfZ/8pyk2yT1KndMHUnSTZIWSnquoGxrSQ9mN159UFLf9l6vk0XnsAq4KCJ2A/YHzpW0W5ljKrcLgBfKHUQn8WPgDxGxK7AnXXi7SBoM/DtQHRGfIJ2Wf0p5o+pwv2Lde+VdAjwcEcOBh7P37crJohOIiDci4qlsfAlpZ7DOvbC6CkmVwL8Cvyh3LOUm6ePAvwC/BIiIFRGxuLxRlV13oHd24W8f4PUyx9OhIuIvQNPrzsYAv87Gfw0c297rdbLoZCRVAXsBT5Y3krL6EXAxsKbcgXQCw4A64OasW+4Xkj5W7qDKJSIWANcCrwJvAO9FxJ/KG1WnMDAi3sjG3wQGtvcKnCw6EUmbA3cBX42I98sdTzlIOgpYGBHTyx1LJ9GddNubn0XEXsCHlKCLYWOR9cWPISXR7YCPSfpCeaPqXCJdD9Hu10Q4WXQSknqQEsWkiPifcsdTRgcAx0iaT3oGymcl3VLekMqqFqiNiPqW5hQa7pnWFR0K/DMi6rIbkP4P8Okyx9QZvCVpEED2ujBn+vXmZNEJSBKpT/qFiPhBueMpp4i4NCIqI6KKdODykYjosr8cI+JN4DVJu2RFhwDPtzLLpu5VYH9JfbL/m0Powgf8C0wF6h8/fTpwb3uvwMmiczgAOI30K3pGNhxZ7qCs0zgfmCTpGdJdmb9X5njKJmthTSHdifpZ0j6sS936Q9JtwOPALpJqJX0RuBo4TNJLpNbX1e2+Xt/uw8zM8rhlYWZmuZwszMwsl5OFmZnlcrIwM7NcThZmZpbLycIsh6TVBac0z5DUbldQS6oqvHuoWWfVvdwBmG0ElkXEyHIHYVZOblmYtZGk+ZL+U9Kzkv4haaesvErSI5KekfSwpO2z8oGS7pY0Mxvqb1NRIenn2TMa/iSpdzb9v2fPOHlG0u1l+phmgJOFWTF6N+mGOrmg7r2I2AP4KeluuQA/AX4dEZ8EJgHXZeXXAX+OiD1J93ealZUPByZExO7AYuD4rPwSYK9sOV8q1YczK4av4DbLIemDiNi8mfL5wGcjYl52I8g3I6KfpLeBQRGxMit/IyL6S6oDKiPio4JlVAEPZg+tQdI3gB4RcZWkPwAfAPcA90TEByX+qGYtcsvCbMNEC+Pr46OC8dU0HEv8V2ACqRUyLXvYj1lZOFmYbZiTC14fz8b/TsOjPscCf83GHwa+DGufMf7xlhYqqRswJCIeBb4BfBxYp3Vj1lH8S8UsX29JMwre/yEi6k+f7ZvdDfYj4NSs7HzSk+2+TnrK3ZlZ+QXAxOwuoatJieMNmlcB3JIlFAHX+XGqVk4+ZmHWRtkxi+qIeLvcsZiVmruhzMwsl1sWZmaWyy0LMzPL5WRhZma5nCzMzCyXk4WZmeVysjAzs1z/HyoboyrZJi3JAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_metric(dfhistory,metric):\r\n",
    "    train_metric=dfhistory[metric]\r\n",
    "    val_metric=dfhistory['val_'+metric]\r\n",
    "    epochs=range(1,len(train_metric)+1)\r\n",
    "    plt.plot(epochs,train_metric,'bo--')\r\n",
    "    plt.plot(epochs,val_metric,'ro--')\r\n",
    "    plt.title('Training and validation '+ metric)\r\n",
    "    plt.xlabel(\"Epochs\")\r\n",
    "    plt.ylabel(metric)\r\n",
    "    plt.legend([\"train_\"+metric, 'val_'+metric])\r\n",
    "    plt.show()\r\n",
    "    \r\n",
    "plot_metric(dfhistory,'loss')\r\n",
    "plot_metric(dfhistory,'auc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "id": "9399ca73-7628-4998-adbd-e10a117717e5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T17:54:21.728679Z",
     "iopub.status.busy": "2022-03-06T17:54:21.727557Z",
     "iopub.status.idle": "2022-03-06T17:54:21.759839Z",
     "shell.execute_reply": "2022-03-06T17:54:21.758659Z",
     "shell.execute_reply.started": "2022-03-06T17:54:21.728590Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Tensor(shape=[400, 1], dtype=float32, place=CPUPlace, stop_gradient=False,\n",
       "       [[0.00698614],\n",
       "        [0.99999368],\n",
       "        [0.00000875],\n",
       "        [0.00000774],\n",
       "        [0.00059250],\n",
       "        [0.55392522],\n",
       "        [0.74556124],\n",
       "        [0.00002774],\n",
       "        [0.00008443],\n",
       "        [0.00005355],\n",
       "        [0.00002804],\n",
       "        [0.15859747],\n",
       "        [0.57526356],\n",
       "        [0.00000017],\n",
       "        [0.01397542],\n",
       "        [0.00002751],\n",
       "        [0.00001771],\n",
       "        [0.99611890],\n",
       "        [0.20293407],\n",
       "        [0.91348392],\n",
       "        [0.99476057],\n",
       "        [0.00162912],\n",
       "        [0.00048150],\n",
       "        [0.00029815],\n",
       "        [0.00116739],\n",
       "        [0.00080913],\n",
       "        [0.00001670],\n",
       "        [0.00103130],\n",
       "        [0.00000017],\n",
       "        [0.00000461],\n",
       "        [0.16735120],\n",
       "        [0.03040477],\n",
       "        [0.03635709],\n",
       "        [0.10908052],\n",
       "        [0.00112515],\n",
       "        [0.04119326],\n",
       "        [0.50048715],\n",
       "        [0.00000039],\n",
       "        [0.00019302],\n",
       "        [0.00003949],\n",
       "        [0.86209637],\n",
       "        [0.06512322],\n",
       "        [0.99157894],\n",
       "        [0.00034263],\n",
       "        [0.00000016],\n",
       "        [0.00032446],\n",
       "        [0.00037528],\n",
       "        [0.06692173],\n",
       "        [0.99408382],\n",
       "        [0.00006297],\n",
       "        [0.00005386],\n",
       "        [0.00000013],\n",
       "        [0.11591765],\n",
       "        [0.00530932],\n",
       "        [0.01025243],\n",
       "        [0.22747663],\n",
       "        [0.00000327],\n",
       "        [0.04511498],\n",
       "        [0.10264020],\n",
       "        [0.13381267],\n",
       "        [0.00000003],\n",
       "        [0.04157181],\n",
       "        [0.00000665],\n",
       "        [0.00000224],\n",
       "        [0.99997032],\n",
       "        [0.97789747],\n",
       "        [0.00000070],\n",
       "        [0.01061553],\n",
       "        [0.00029709],\n",
       "        [0.04917601],\n",
       "        [0.00000495],\n",
       "        [0.00001529],\n",
       "        [0.00006696],\n",
       "        [0.00130523],\n",
       "        [0.02542407],\n",
       "        [0.33486718],\n",
       "        [0.94728172],\n",
       "        [0.96759498],\n",
       "        [0.00000662],\n",
       "        [0.00083532],\n",
       "        [0.00198941],\n",
       "        [0.15853325],\n",
       "        [0.99960858],\n",
       "        [0.00381799],\n",
       "        [0.00018856],\n",
       "        [0.02778450],\n",
       "        [0.00000568],\n",
       "        [0.00189981],\n",
       "        [0.00000153],\n",
       "        [0.12668218],\n",
       "        [0.13392943],\n",
       "        [0.00067402],\n",
       "        [0.84998626],\n",
       "        [0.04872032],\n",
       "        [0.01419438],\n",
       "        [0.95523739],\n",
       "        [0.08299810],\n",
       "        [0.00000115],\n",
       "        [0.99437571],\n",
       "        [0.01472239],\n",
       "        [0.00000029],\n",
       "        [0.00728839],\n",
       "        [0.00863015],\n",
       "        [0.00000945],\n",
       "        [0.78405124],\n",
       "        [0.00000812],\n",
       "        [0.23164046],\n",
       "        [0.00003361],\n",
       "        [0.00055185],\n",
       "        [0.00062357],\n",
       "        [0.00005563],\n",
       "        [0.98631269],\n",
       "        [0.77214867],\n",
       "        [0.87438059],\n",
       "        [0.00094726],\n",
       "        [0.00128387],\n",
       "        [0.99966705],\n",
       "        [0.85529995],\n",
       "        [0.99465936],\n",
       "        [0.01128775],\n",
       "        [0.00000503],\n",
       "        [0.00026301],\n",
       "        [0.00468276],\n",
       "        [0.00074982],\n",
       "        [0.98022974],\n",
       "        [0.00021256],\n",
       "        [0.93828744],\n",
       "        [0.00002073],\n",
       "        [0.00706731],\n",
       "        [0.99964488],\n",
       "        [0.00079850],\n",
       "        [0.00340911],\n",
       "        [0.00003029],\n",
       "        [0.00000529],\n",
       "        [0.00128528],\n",
       "        [0.00141162],\n",
       "        [0.00002932],\n",
       "        [0.00004903],\n",
       "        [0.01833274],\n",
       "        [0.00186182],\n",
       "        [0.01067017],\n",
       "        [0.00189622],\n",
       "        [0.00759995],\n",
       "        [0.00974044],\n",
       "        [0.99837589],\n",
       "        [0.00055820],\n",
       "        [0.00002814],\n",
       "        [0.99919444],\n",
       "        [0.00000331],\n",
       "        [0.01283903],\n",
       "        [0.00224520],\n",
       "        [0.81208867],\n",
       "        [0.00530503],\n",
       "        [0.25850940],\n",
       "        [0.96876270],\n",
       "        [0.00325632],\n",
       "        [0.00804867],\n",
       "        [0.91198486],\n",
       "        [0.00004647],\n",
       "        [0.06818192],\n",
       "        [0.89116704],\n",
       "        [0.00000140],\n",
       "        [0.97169262],\n",
       "        [0.06782891],\n",
       "        [0.95108902],\n",
       "        [0.24589700],\n",
       "        [0.00545287],\n",
       "        [0.00000081],\n",
       "        [0.18845138],\n",
       "        [0.00028436],\n",
       "        [0.00000708],\n",
       "        [0.00126399],\n",
       "        [0.00031768],\n",
       "        [0.00005568],\n",
       "        [0.00171518],\n",
       "        [0.99989808],\n",
       "        [0.00006541],\n",
       "        [0.02417973],\n",
       "        [0.00225583],\n",
       "        [0.00000059],\n",
       "        [0.65112209],\n",
       "        [0.06535023],\n",
       "        [0.85523766],\n",
       "        [0.98514682],\n",
       "        [0.00110180],\n",
       "        [0.02162250],\n",
       "        [0.01716501],\n",
       "        [0.00005870],\n",
       "        [0.00000028],\n",
       "        [0.05413519],\n",
       "        [0.00271012],\n",
       "        [0.94621611],\n",
       "        [0.00000615],\n",
       "        [0.17586544],\n",
       "        [0.00067941],\n",
       "        [0.00261440],\n",
       "        [0.00141161],\n",
       "        [0.99373573],\n",
       "        [0.00235063],\n",
       "        [0.40164161],\n",
       "        [0.99960262],\n",
       "        [0.00036851],\n",
       "        [0.00009470],\n",
       "        [0.00000724],\n",
       "        [0.52651733],\n",
       "        [0.01193550],\n",
       "        [0.00002632],\n",
       "        [0.01032089],\n",
       "        [0.76069492],\n",
       "        [0.04643028],\n",
       "        [0.00028911],\n",
       "        [0.99801743],\n",
       "        [0.28478196],\n",
       "        [0.02541153],\n",
       "        [0.00000439],\n",
       "        [0.00040329],\n",
       "        [0.00173613],\n",
       "        [0.03840018],\n",
       "        [0.00000599],\n",
       "        [0.70641828],\n",
       "        [0.07672497],\n",
       "        [0.33973667],\n",
       "        [0.00038402],\n",
       "        [0.02989760],\n",
       "        [0.29367375],\n",
       "        [0.00000201],\n",
       "        [0.10259436],\n",
       "        [0.05369290],\n",
       "        [0.00254449],\n",
       "        [0.00432377],\n",
       "        [0.14857696],\n",
       "        [0.99921548],\n",
       "        [0.00152014],\n",
       "        [0.98031759],\n",
       "        [0.42144439],\n",
       "        [0.94969666],\n",
       "        [0.99944383],\n",
       "        [0.00032673],\n",
       "        [0.43077433],\n",
       "        [0.00020052],\n",
       "        [0.99824500],\n",
       "        [0.00117800],\n",
       "        [0.00113616],\n",
       "        [0.04312274],\n",
       "        [0.00023393],\n",
       "        [0.00012615],\n",
       "        [0.00000733],\n",
       "        [0.00000185],\n",
       "        [0.01144225],\n",
       "        [0.97697693],\n",
       "        [0.00370012],\n",
       "        [0.00131340],\n",
       "        [0.00000157],\n",
       "        [0.02920556],\n",
       "        [0.00001866],\n",
       "        [0.33885419],\n",
       "        [0.00270135],\n",
       "        [0.00007196],\n",
       "        [0.00000452],\n",
       "        [0.70394462],\n",
       "        [0.17566633],\n",
       "        [0.00009218],\n",
       "        [0.88777941],\n",
       "        [0.00003145],\n",
       "        [0.01795338],\n",
       "        [0.00028099],\n",
       "        [0.00503764],\n",
       "        [0.00000118],\n",
       "        [0.00090557],\n",
       "        [0.36434343],\n",
       "        [0.11675984],\n",
       "        [0.00088319],\n",
       "        [0.00748920],\n",
       "        [0.63681042],\n",
       "        [0.00005655],\n",
       "        [0.02324953],\n",
       "        [0.02339749],\n",
       "        [0.55304056],\n",
       "        [0.23493387],\n",
       "        [0.00002942],\n",
       "        [0.20684014],\n",
       "        [0.20380607],\n",
       "        [0.00000391],\n",
       "        [0.00000068],\n",
       "        [0.73458886],\n",
       "        [0.77821201],\n",
       "        [0.00000166],\n",
       "        [0.14301080],\n",
       "        [0.00092198],\n",
       "        [0.02198396],\n",
       "        [0.00136311],\n",
       "        [0.01840463],\n",
       "        [0.97541624],\n",
       "        [0.00439648],\n",
       "        [0.00000689],\n",
       "        [0.00000242],\n",
       "        [0.00018858],\n",
       "        [0.00015886],\n",
       "        [0.00112421],\n",
       "        [0.00424471],\n",
       "        [0.00008801],\n",
       "        [0.00467102],\n",
       "        [0.76257759],\n",
       "        [0.00037815],\n",
       "        [0.99072367],\n",
       "        [0.25442240],\n",
       "        [0.00084000],\n",
       "        [0.00277778],\n",
       "        [0.00000058],\n",
       "        [0.00269234],\n",
       "        [0.00005348],\n",
       "        [0.00006821],\n",
       "        [0.00000010],\n",
       "        [0.68406433],\n",
       "        [0.46138516],\n",
       "        [0.97209364],\n",
       "        [0.00040926],\n",
       "        [0.00009235],\n",
       "        [0.55935335],\n",
       "        [0.00197447],\n",
       "        [0.96632260],\n",
       "        [0.95248592],\n",
       "        [0.08993820],\n",
       "        [0.00008541],\n",
       "        [0.61808830],\n",
       "        [0.50855476],\n",
       "        [0.32514498],\n",
       "        [0.00008533],\n",
       "        [0.00001019],\n",
       "        [0.00000127],\n",
       "        [0.00007910],\n",
       "        [0.00000008],\n",
       "        [0.19278580],\n",
       "        [0.92339307],\n",
       "        [0.60193914],\n",
       "        [0.21617578],\n",
       "        [0.01062161],\n",
       "        [0.00095136],\n",
       "        [0.00004197],\n",
       "        [0.24377753],\n",
       "        [0.00039676],\n",
       "        [0.96583796],\n",
       "        [0.00024352],\n",
       "        [0.00025974],\n",
       "        [0.00024411],\n",
       "        [0.00040577],\n",
       "        [0.00000386],\n",
       "        [0.29900518],\n",
       "        [0.51334435],\n",
       "        [0.00704208],\n",
       "        [0.00000266],\n",
       "        [0.52695680],\n",
       "        [0.99921203],\n",
       "        [0.99940479],\n",
       "        [0.00047987],\n",
       "        [0.96201885],\n",
       "        [0.39437008],\n",
       "        [0.99914992],\n",
       "        [0.77732146],\n",
       "        [0.99760109],\n",
       "        [0.94911194],\n",
       "        [0.00423393],\n",
       "        [0.08721106],\n",
       "        [0.00083633],\n",
       "        [0.16368085],\n",
       "        [0.00002321],\n",
       "        [0.00000711],\n",
       "        [0.92166531],\n",
       "        [0.04013541],\n",
       "        [0.00039924],\n",
       "        [0.03055409],\n",
       "        [0.00000552],\n",
       "        [0.19223665],\n",
       "        [0.00001669],\n",
       "        [0.00536732],\n",
       "        [0.19563806],\n",
       "        [0.00608837],\n",
       "        [0.00000245],\n",
       "        [0.40704626],\n",
       "        [0.00366030],\n",
       "        [0.00000056],\n",
       "        [0.22808990],\n",
       "        [0.07956171],\n",
       "        [0.99847025],\n",
       "        [0.08798286],\n",
       "        [0.99952555],\n",
       "        [0.00083798],\n",
       "        [0.00000340],\n",
       "        [0.00764355],\n",
       "        [0.00054901],\n",
       "        [0.03183319],\n",
       "        [0.10659497],\n",
       "        [0.00000063],\n",
       "        [0.97251850],\n",
       "        [0.29046974],\n",
       "        [0.00040539],\n",
       "        [0.46013683],\n",
       "        [0.33089533],\n",
       "        [0.01305736],\n",
       "        [0.05919992]])"
      ]
     },
     "execution_count": 216,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#预测\r\n",
    "testx=paddle.to_tensor(test_x)\r\n",
    "testx = paddle.cast(testx, dtype='float32')\r\n",
    "y_pred_probs=model(testx)\r\n",
    "y_pred_probs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6407dd58-189b-40fb-95da-ea43d266bd13",
   "metadata": {},
   "source": [
    "# 六、总结与升华\n",
    "\n",
    "deepFM可以自己寻找特征再也不需要人工找特征，但是，也有可能找到的特征是噪音"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a97527af-0d43-4adb-9cb7-9956671e692f",
   "metadata": {},
   "source": [
    "# 七、个人总结\n",
    "\n",
    "第一次利用paddle算法来做，若有错误之处，欢迎大家指出！\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
